[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:50 dist-test-slave-bwpp krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:50 dist-test-slave-bwpp krb5kdc[399](info): set up 2 sockets
Jun 20 12:24:50 dist-test-slave-bwpp krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 12:24:51.975148 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:52 dist-test-slave-bwpp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:52.096040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38535
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:52.212925 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:52.213205 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:52.213245 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:52.217235 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:52.217286 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:52.217306 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:52.217322 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:52.217337 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:52.222527 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:52.223544 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:52.224538 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:24:52.233197 412 server_base.cc:1034] running on GCE node
W20260620 12:24:52.233165 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:52.233246 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:52.233335 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:52.234025 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:52.235175 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:52.236331 412 hybrid_clock.cc:648] HybridClock initialized: now 1781958292236285 us; error 59 us; skew 500 ppm
Jun 20 12:24:52 dist-test-slave-bwpp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:52.239001 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:52.239836 412 webserver.cc:460] Webserver started at http://127.0.98.62:44139/ using document root <none> and password file <none>
I20260620 12:24:52.240392 412 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:52.240450 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:52.240700 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:52.242833 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "71b44839e9fd424e814dd6c0b1194313"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "211857859fe06da62937c24fb8667f5d"
server_key_iv: "1cfb5808b3f9c16893f9daafa4e2ec97"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.243407 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "71b44839e9fd424e814dd6c0b1194313"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "211857859fe06da62937c24fb8667f5d"
server_key_iv: "1cfb5808b3f9c16893f9daafa4e2ec97"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.247041 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:24:52.249334 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:52.250535 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:52.250722 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "71b44839e9fd424e814dd6c0b1194313"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "211857859fe06da62937c24fb8667f5d"
server_key_iv: "1cfb5808b3f9c16893f9daafa4e2ec97"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.250840 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:52.268169 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:52.271085 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:52.271311 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:52.278469 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38535
I20260620 12:24:52.278489 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38535 every 8 connection(s)
I20260620 12:24:52.279495 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:52.282182 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:52.282401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 412
I20260620 12:24:52.282574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:52.283869 392 external_mini_cluster.cc:1428] Setting key 0b327dafb5ca478c031de865924c5577
I20260620 12:24:52.288252 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313: Bootstrap starting.
I20260620 12:24:52.291272 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:52.292200 480 log.cc:824] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:52.294479 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313: No bootstrap required, opened a new log
I20260620 12:24:52.298857 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b44839e9fd424e814dd6c0b1194313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38535 } }
I20260620 12:24:52.299130 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:52.299208 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b44839e9fd424e814dd6c0b1194313, State: Initialized, Role: FOLLOWER
I20260620 12:24:52.299705 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b44839e9fd424e814dd6c0b1194313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38535 } }
I20260620 12:24:52.299857 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:24:52.299933 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:24:52.300036 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:52.301340 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b44839e9fd424e814dd6c0b1194313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38535 } }
I20260620 12:24:52.301693 480 leader_election.cc:302] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71b44839e9fd424e814dd6c0b1194313; no voters:
I20260620 12:24:52.302037 480 leader_election.cc:288] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:24:52.302110 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:52.302312 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [term 1 LEADER]: Becoming Leader. State: Replica: 71b44839e9fd424e814dd6c0b1194313, State: Running, Role: LEADER
I20260620 12:24:52.302625 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b44839e9fd424e814dd6c0b1194313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38535 } }
I20260620 12:24:52.303418 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 12:24:52 dist-test-slave-bwpp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:52.305538 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71b44839e9fd424e814dd6c0b1194313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b44839e9fd424e814dd6c0b1194313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38535 } } }
I20260620 12:24:52.305657 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:52.306133 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b44839e9fd424e814dd6c0b1194313. Latest consensus state: current_term: 1 leader_uuid: "71b44839e9fd424e814dd6c0b1194313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b44839e9fd424e814dd6c0b1194313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38535 } } }
I20260620 12:24:52.306236 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313 [sys.catalog]: This master's current role is: LEADER
W20260620 12:24:52.308995 499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:52.309069 499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:52.309128 501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:52.309921 501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:52.311592 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:52.286922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.0.98.62:38535)
0620 12:24:52.287428 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:52.287451 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:52.288467 (+ 1016us) server_negotiation.cc:407] Connection header received
0620 12:24:52.292907 (+ 4440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:52.292932 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:52.293279 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:52.293550 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:52.296563 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.297527 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.299237 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.299754 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.299946 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:52.305129 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:52.305139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:52.305160 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:52.305201 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:52.307625 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:52.308321 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:52.308328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:52.308334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:52.308488 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:52.308907 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:52.308911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:52.308913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:52.309430 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:52.309632 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:52.310330 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 12:24:52.310621 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":137,"threads_started":1}
I20260620 12:24:52.315820 501 catalog_manager.cc:1347] Generated new cluster ID: 05a7e3aca061498e9e8115218d14e696
I20260620 12:24:52.315894 501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:52.327059 501 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:52.328027 501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:52.338354 501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71b44839e9fd424e814dd6c0b1194313: Generated new TSK 0
I20260620 12:24:52.339159 501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:24:52.371896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38535
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:52.483321 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:52.483642 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:52.483726 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:52.487828 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:52.487926 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:52.488023 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:24:52.493431 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:52.494571 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:52.495674 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:52.505340 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:52.505340 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:52.505360 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:52.505854 506 server_base.cc:1034] running on GCE node
I20260620 12:24:52.506258 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:52.506980 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:52.508162 506 hybrid_clock.cc:648] HybridClock initialized: now 1781958292508149 us; error 26 us; skew 500 ppm
Jun 20 12:24:52 dist-test-slave-bwpp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:52.511009 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:24:52.511898 506 webserver.cc:460] Webserver started at http://127.0.98.1:42011/ using document root <none> and password file <none>
I20260620 12:24:52.512612 506 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:52.512714 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:52.512943 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:52.515098 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6e7f7fe695f4c538255c8d37be55ab2"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "af2f85732748f1df93c503a969ad5d9b"
server_key_iv: "40e5c192f556c1d3c7ab10237b78b515"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.515686 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6e7f7fe695f4c538255c8d37be55ab2"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "af2f85732748f1df93c503a969ad5d9b"
server_key_iv: "40e5c192f556c1d3c7ab10237b78b515"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.519501 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:24:52.521880 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:52.523196 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:52.523341 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "d6e7f7fe695f4c538255c8d37be55ab2"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "af2f85732748f1df93c503a969ad5d9b"
server_key_iv: "40e5c192f556c1d3c7ab10237b78b515"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.523463 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:52.564386 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:52.567677 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:52.567919 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:52.568562 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:24:52.569463 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:52.569543 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:52.569626 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:52.569675 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:52.579195 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33049
I20260620 12:24:52.579217 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33049 every 8 connection(s)
I20260620 12:24:52.580230 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:24:52.589087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 506
I20260620 12:24:52.589182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:24:52.589435 392 external_mini_cluster.cc:1428] Setting key 8505af590d62dbf5b9ef2983438777b1
Jun 20 12:24:52 dist-test-slave-bwpp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:52.595748 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:52.582166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36371 (local address 127.0.98.62:38535)
0620 12:24:52.582313 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:52.582316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:52.583640 (+ 1324us) server_negotiation.cc:407] Connection header received
0620 12:24:52.585029 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:52.585032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:52.585096 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:52.585205 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:52.586412 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.587020 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.588180 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.588417 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.588531 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:52.591713 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:52.591717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:52.591719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:52.591743 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:52.593588 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:52.594125 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:52.594129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:52.594131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:52.594215 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:52.594572 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:52.594576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:52.594578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:52.594824 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:52.594946 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:52.595453 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 12:24:52.595581 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:24:52.596668 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:52.582605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38535 (local address 127.0.98.1:36371)
0620 12:24:52.583076 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:52.583122 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:52.584794 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:52.585390 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:52.585398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:52.585769 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:52.586246 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:52.586260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.587149 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.587155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:52.588049 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:52.588059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.588573 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.588577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:52.588748 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:52.589598 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:52.589621 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:52.591545 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:52.593723 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:52.593728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:52.593741 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:52.594012 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:52.594315 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:52.594318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:52.594320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:52.594462 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:52.594945 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:52.594952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:52.595209 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 12:24:52.595414 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 12:24:52.595781 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260620 12:24:52.598076 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38535
I20260620 12:24:52.598325 635 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:52.598873 635 heartbeater.cc:511] Master 127.0.98.62:38535 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:24:52.600510 444 ts_manager.cc:194] Registered new tserver with Master: d6e7f7fe695f4c538255c8d37be55ab2 (127.0.98.1:33049)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:24:52.601823 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:24:52.617177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38535
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:52.728308 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:52.728578 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:52.728623 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:52.732596 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:52.732653 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:52.732714 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:24:52.737812 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:52.738821 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:52.739845 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:52.749444 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:52.749485 640 server_base.cc:1034] running on GCE node
W20260620 12:24:52.749547 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:52.749439 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:52.750095 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:52.750835 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:52.752009 640 hybrid_clock.cc:648] HybridClock initialized: now 1781958292751993 us; error 30 us; skew 500 ppm
Jun 20 12:24:52 dist-test-slave-bwpp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:52.754952 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:24:52.755877 640 webserver.cc:460] Webserver started at http://127.0.98.2:34435/ using document root <none> and password file <none>
I20260620 12:24:52.756410 640 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:52.756469 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:52.756646 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:52.758785 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "d997aa3d542a41779f6f1f64fe1cba18"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "9f11c16707e608ac93ee826041b8ea7e"
server_key_iv: "8c56fa2b9e9e25a40f798945ed99cf92"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.759402 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d997aa3d542a41779f6f1f64fe1cba18"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "9f11c16707e608ac93ee826041b8ea7e"
server_key_iv: "8c56fa2b9e9e25a40f798945ed99cf92"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.763433 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:24:52.766098 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:52.767426 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:24:52.767596 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "d997aa3d542a41779f6f1f64fe1cba18"
format_stamp: "Formatted at 2026-06-20 12:24:52 on dist-test-slave-bwpp"
server_key: "9f11c16707e608ac93ee826041b8ea7e"
server_key_iv: "8c56fa2b9e9e25a40f798945ed99cf92"
server_key_version: "encryptionkey@0"
I20260620 12:24:52.767727 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:52.822336 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:52.826040 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:52.826256 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:52.827001 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:24:52.827924 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:52.827994 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:52.828060 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:52.828104 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:52.838052 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35079
I20260620 12:24:52.838068 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35079 every 8 connection(s)
I20260620 12:24:52.839068 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:24:52.843675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 640
I20260620 12:24:52.843789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:24:52.844059 392 external_mini_cluster.cc:1428] Setting key b53beb4d2dcc2286b9c4a84a6b92c054
Jun 20 12:24:52 dist-test-slave-bwpp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:24:52.853147 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:52.840944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48393 (local address 127.0.98.62:38535)
0620 12:24:52.841107 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:52.841113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:52.842003 (+ 890us) server_negotiation.cc:407] Connection header received
0620 12:24:52.843314 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:52.843317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:52.843365 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:52.843433 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:52.844909 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.845509 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.846718 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.846948 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.847056 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:52.849501 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:52.849504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:52.849506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:52.849528 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:52.851063 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:52.851623 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:52.851627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:52.851629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:52.851713 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:52.852061 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:52.852066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:52.852068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:52.852251 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:52.852358 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:52.852857 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 12:24:52.852980 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:24:52.854087 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:52.841368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38535 (local address 127.0.98.2:48393)
0620 12:24:52.841826 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:52.841882 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:52.843100 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:52.843645 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:52.843656 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:52.844183 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:52.844744 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:52.844757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.845661 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.845667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:52.846562 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:52.846571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:52.847106 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:52.847110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:52.847275 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:52.847864 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:52.847880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:52.849314 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:52.851197 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:52.851203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:52.851219 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:52.851485 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:52.851810 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:52.851814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:52.851816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:52.851955 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:52.852357 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:52.852364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:52.852623 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 12:24:52.852821 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 12:24:52.853159 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:24:52.855571 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38535
I20260620 12:24:52.855815 769 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:52.856313 769 heartbeater.cc:511] Master 127.0.98.62:38535 requested a full tablet report, sending...
I20260620 12:24:52.857481 444 ts_manager.cc:194] Registered new tserver with Master: d997aa3d542a41779f6f1f64fe1cba18 (127.0.98.2:35079)
I20260620 12:24:52.858071 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:24:52.870020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38535
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:52.986303 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:52.986596 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:52.986642 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:52.990813 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:52.990883 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:52.990947 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:24:52.996335 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:52.997376 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:52.998348 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:53.008484 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:53.008492 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:53.008587 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:53.009017 774 server_base.cc:1034] running on GCE node
I20260620 12:24:53.009429 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:53.010129 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:53.011312 774 hybrid_clock.cc:648] HybridClock initialized: now 1781958293011302 us; error 33 us; skew 500 ppm
Jun 20 12:24:53 dist-test-slave-bwpp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:53.014108 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:24:53.014997 774 webserver.cc:460] Webserver started at http://127.0.98.3:38039/ using document root <none> and password file <none>
I20260620 12:24:53.015518 774 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:53.015574 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:53.015738 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:53.017752 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "acfa9dfe5dbe4977bbbcb54429a1fecf"
format_stamp: "Formatted at 2026-06-20 12:24:53 on dist-test-slave-bwpp"
server_key: "e3eb6a89ffd8431821b763437fa8d00f"
server_key_iv: "3a447a63cbf8da832611a77babf18c65"
server_key_version: "encryptionkey@0"
I20260620 12:24:53.018266 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "acfa9dfe5dbe4977bbbcb54429a1fecf"
format_stamp: "Formatted at 2026-06-20 12:24:53 on dist-test-slave-bwpp"
server_key: "e3eb6a89ffd8431821b763437fa8d00f"
server_key_iv: "3a447a63cbf8da832611a77babf18c65"
server_key_version: "encryptionkey@0"
I20260620 12:24:53.022099 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:24:53.024691 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:53.025923 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:53.026070 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "acfa9dfe5dbe4977bbbcb54429a1fecf"
format_stamp: "Formatted at 2026-06-20 12:24:53 on dist-test-slave-bwpp"
server_key: "e3eb6a89ffd8431821b763437fa8d00f"
server_key_iv: "3a447a63cbf8da832611a77babf18c65"
server_key_version: "encryptionkey@0"
I20260620 12:24:53.026188 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:53.053772 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:53.057334 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:53.057500 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:53.058039 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:24:53.058887 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:53.058936 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:53.058977 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:53.058993 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:53.068879 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42497
I20260620 12:24:53.068956 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42497 every 8 connection(s)
I20260620 12:24:53.069850 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:24:53.077101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 774
I20260620 12:24:53.077246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:24:53.077526 392 external_mini_cluster.cc:1428] Setting key c9c140a3d5f269320b9d49695582fa25
Jun 20 12:24:53 dist-test-slave-bwpp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:53.083881 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:53.071872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50957 (local address 127.0.98.62:38535)
0620 12:24:53.072025 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:53.072028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:53.072935 (+ 907us) server_negotiation.cc:407] Connection header received
0620 12:24:53.074309 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:53.074313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:53.074367 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:53.074477 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:53.075730 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.076260 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.077503 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.077766 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.077878 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:53.080749 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:53.080752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:53.080754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:53.080778 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:53.082052 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:53.082524 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:53.082526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:53.082527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:53.082597 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:53.082947 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:53.082949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:53.082950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:53.083121 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:53.083198 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:53.083656 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 12:24:53.083761 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:24:53.084721 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:53.072329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38535 (local address 127.0.98.3:50957)
0620 12:24:53.072783 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:53.072826 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:53.074095 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:53.074706 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:53.074715 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:53.075098 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:53.075580 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:53.075593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.076382 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.076386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:53.077379 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:53.077388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.077917 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.077921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:53.078107 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:53.078832 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:53.078849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:53.080604 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:53.082155 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:53.082159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:53.082171 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:53.082429 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:53.082720 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:53.082723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:53.082724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:53.082854 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:53.083208 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:53.083213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:53.083447 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 12:24:53.083595 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 12:24:53.083850 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260620 12:24:53.085814 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38535
I20260620 12:24:53.086069 903 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:53.086592 903 heartbeater.cc:511] Master 127.0.98.62:38535 requested a full tablet report, sending...
I20260620 12:24:53.087639 444 ts_manager.cc:194] Registered new tserver with Master: acfa9dfe5dbe4977bbbcb54429a1fecf (127.0.98.3:42497)
I20260620 12:24:53.088213 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50957
I20260620 12:24:53.091982 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:24:53 dist-test-slave-bwpp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:24:53 dist-test-slave-bwpp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:24:53.118023 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:53.108466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.0.98.1:33049)
0620 12:24:53.108805 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:53.108812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:53.108835 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:24:53.108959 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:53.108965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:53.109175 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:53.109298 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:53.109992 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.110803 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.111619 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.111975 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.112062 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:53.114578 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:53.114587 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:53.114599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:53.114624 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:53.116369 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:53.116796 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:53.116801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:53.116802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:53.116883 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:53.117111 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:53.117114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:53.117115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:53.117342 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:53.117498 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:53.117649 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 12:24:53.117808 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":140,"threads_started":1}
W20260620 12:24:53.119093 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35432
Jun 20 12:24:53 dist-test-slave-bwpp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:24:53 dist-test-slave-bwpp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:24:53.160230 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:53.151324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.0.98.1:33049)
0620 12:24:53.151469 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:53.151475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:53.151680 (+ 205us) server_negotiation.cc:407] Connection header received
0620 12:24:53.151849 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:53.151852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:53.151924 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:53.152017 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:53.152753 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.153270 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.154177 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:53.154406 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:53.154519 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:53.157246 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:53.157250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:53.157254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:53.157286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:53.158528 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:53.158983 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:53.158987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:53.158989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:53.159071 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:53.159355 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:53.159358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:53.159360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:53.159548 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:53.159703 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:53.159939 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 12:24:53.160060 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:24:53.161278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 506
I20260620 12:24:53.167493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 640
I20260620 12:24:53.175439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 774
I20260620 12:24:53.181245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 412
2026-06-20T12:24:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2261 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:54 dist-test-slave-bwpp krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:54 dist-test-slave-bwpp krb5kdc[917](info): set up 2 sockets
Jun 20 12:24:54 dist-test-slave-bwpp krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 12:24:55.050453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.846s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:55 dist-test-slave-bwpp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:55.173415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:55.283190 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:55.283463 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:55.283502 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:55.287415 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:55.287478 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:55.287498 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:55.287515 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:55.287530 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:55.292905 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:55.293982 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:55.294986 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:55.303344 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:55.303413 930 server_base.cc:1034] running on GCE node
W20260620 12:24:55.303431 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:55.303584 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:55.304160 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:55.305310 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:55.306499 930 hybrid_clock.cc:648] HybridClock initialized: now 1781958295306484 us; error 33 us; skew 500 ppm
Jun 20 12:24:55 dist-test-slave-bwpp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:55.309245 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:55.310163 930 webserver.cc:460] Webserver started at http://127.0.98.62:34127/ using document root <none> and password file <none>
I20260620 12:24:55.310787 930 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:55.310874 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:55.311108 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:55.313122 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "055640002c53471fbadf209e0a43b8f2"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "cec780f597194b5cd11a1766b0225546"
server_key_iv: "c4750aef1b5b03f204141abcfa265e3c"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.313678 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "055640002c53471fbadf209e0a43b8f2"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "cec780f597194b5cd11a1766b0225546"
server_key_iv: "c4750aef1b5b03f204141abcfa265e3c"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.317396 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:24:55.319806 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:55.320969 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:24:55.321101 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "055640002c53471fbadf209e0a43b8f2"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "cec780f597194b5cd11a1766b0225546"
server_key_iv: "c4750aef1b5b03f204141abcfa265e3c"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.321214 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:55.348189 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:55.351289 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:55.351504 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:55.358747 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36061
I20260620 12:24:55.358769 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36061 every 8 connection(s)
I20260620 12:24:55.359743 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:55.360010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 930
I20260620 12:24:55.360136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:55.360430 392 external_mini_cluster.cc:1428] Setting key e4edaadfbd336176fb303d4c9a087f6c
I20260620 12:24:55.363029 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:55.368474 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2: Bootstrap starting.
Jun 20 12:24:55 dist-test-slave-bwpp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:55.370915 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:55.371621 998 log.cc:824] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:55.373550 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2: No bootstrap required, opened a new log
I20260620 12:24:55.374015 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:55.361831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.0.98.62:36061)
0620 12:24:55.362275 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:55.362290 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:55.362326 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:24:55.363025 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:55.363042 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:55.363406 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:55.363750 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:55.364499 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.365483 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.366306 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.366839 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.367001 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:55.369517 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:55.369522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:55.369535 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:55.369560 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:55.371282 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:55.371698 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:55.371709 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:55.371713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:55.371865 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:55.372147 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:55.372150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:55.372152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:55.372536 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:55.372715 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:55.373005 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 12:24:55.373271 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":154,"threads_started":1}
I20260620 12:24:55.377182 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055640002c53471fbadf209e0a43b8f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260620 12:24:55.377451 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:55.377523 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055640002c53471fbadf209e0a43b8f2, State: Initialized, Role: FOLLOWER
I20260620 12:24:55.377987 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055640002c53471fbadf209e0a43b8f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260620 12:24:55.378154 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:24:55.378234 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:24:55.378312 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:55.379231 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055640002c53471fbadf209e0a43b8f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260620 12:24:55.379535 998 leader_election.cc:302] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 055640002c53471fbadf209e0a43b8f2; no voters:
I20260620 12:24:55.379794 998 leader_election.cc:288] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:24:55.379985 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:55.380245 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [term 1 LEADER]: Becoming Leader. State: Replica: 055640002c53471fbadf209e0a43b8f2, State: Running, Role: LEADER
I20260620 12:24:55.380532 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055640002c53471fbadf209e0a43b8f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260620 12:24:55.380998 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:55.382088 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 055640002c53471fbadf209e0a43b8f2. Latest consensus state: current_term: 1 leader_uuid: "055640002c53471fbadf209e0a43b8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055640002c53471fbadf209e0a43b8f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } } }
I20260620 12:24:55.382208 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:55.382521 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "055640002c53471fbadf209e0a43b8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055640002c53471fbadf209e0a43b8f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } } }
I20260620 12:24:55.382599 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:55.382701 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:55.385727 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:55.391084 1012 catalog_manager.cc:1347] Generated new cluster ID: 171180ca194c4dbc888468d34ac517ea
I20260620 12:24:55.391160 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:55.420253 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:55.421701 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:55.431910 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 055640002c53471fbadf209e0a43b8f2: Generated new TSK 0
I20260620 12:24:55.432636 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:24:55.461639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:55.573604 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:55.573938 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:55.574013 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:55.578147 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:55.578243 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:55.578341 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:24:55.583986 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:55.585131 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:55.586231 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:55.596755 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:55.596724 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:55.596812 1024 server_base.cc:1034] running on GCE node
W20260620 12:24:55.596737 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:55.597399 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:55.598093 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:55.599249 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781958295599234 us; error 32 us; skew 500 ppm
Jun 20 12:24:55 dist-test-slave-bwpp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:55.602100 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:24:55.603209 1024 webserver.cc:460] Webserver started at http://127.0.98.1:37613/ using document root <none> and password file <none>
I20260620 12:24:55.603845 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:55.603943 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:55.604239 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:55.606570 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "47fb08479c274196a1585e6521fa997e"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "9d518a23192561d39fc362b0500e8ee1"
server_key_iv: "c59aa7b15ad8b5ad846975bdfdd40ced"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.607241 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47fb08479c274196a1585e6521fa997e"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "9d518a23192561d39fc362b0500e8ee1"
server_key_iv: "c59aa7b15ad8b5ad846975bdfdd40ced"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.611590 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:24:55.614382 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:55.615747 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:24:55.615898 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "47fb08479c274196a1585e6521fa997e"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "9d518a23192561d39fc362b0500e8ee1"
server_key_iv: "c59aa7b15ad8b5ad846975bdfdd40ced"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.616019 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:55.627067 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:55.630120 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:55.630317 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:55.630990 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:24:55.631830 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:55.631899 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:55.631982 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:55.632026 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:55.641446 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38347
I20260620 12:24:55.641467 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38347 every 8 connection(s)
I20260620 12:24:55.642364 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:24:55.648440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 1024
I20260620 12:24:55.648557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:24:55.648856 392 external_mini_cluster.cc:1428] Setting key b77ba009330f4bf9b5e9489a7a24a4cb
Jun 20 12:24:55 dist-test-slave-bwpp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:55.657094 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:55.644304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57095 (local address 127.0.98.62:36061)
0620 12:24:55.644448 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:55.644451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:55.645288 (+ 837us) server_negotiation.cc:407] Connection header received
0620 12:24:55.646694 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:55.646697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:55.646749 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:55.646829 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:55.647992 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.648533 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.650126 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.650458 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.650553 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:55.653430 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:55.653434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:55.653436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:55.653458 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:55.655013 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:55.655537 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:55.655542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:55.655544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:55.655621 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:55.655956 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:55.655961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:55.655963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:55.656154 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:55.656265 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:55.656776 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 12:24:55.656910 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:24:55.658001 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:55.644707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36061 (local address 127.0.98.1:57095)
0620 12:24:55.645149 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:55.645194 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:55.646470 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:55.647000 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:55.647007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:55.647378 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:55.647850 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:55.647864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.648703 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.648708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:55.649900 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:55.649911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.650620 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.650624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:55.650878 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:55.651604 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:55.651627 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:55.653256 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:55.655133 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:55.655139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:55.655156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:55.655425 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:55.655717 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:55.655720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:55.655722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:55.655865 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:55.656275 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:55.656284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:55.656545 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 12:24:55.656737 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 12:24:55.657076 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260620 12:24:55.659255 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36061
I20260620 12:24:55.659530 1153 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:55.660137 1153 heartbeater.cc:511] Master 127.0.98.62:36061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:24:55.661808 962 ts_manager.cc:194] Registered new tserver with Master: 47fb08479c274196a1585e6521fa997e (127.0.98.1:38347)
I20260620 12:24:55.663023 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:24:55.677529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:55.789740 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:55.790045 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:55.790091 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:55.794248 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:55.794310 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:55.794370 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:24:55.799685 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:55.800724 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:55.801738 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:55.811494 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:55.811468 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:55.811468 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:55.811612 1158 server_base.cc:1034] running on GCE node
I20260620 12:24:55.812207 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:55.812855 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:55.814034 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781958295814018 us; error 29 us; skew 500 ppm
Jun 20 12:24:55 dist-test-slave-bwpp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:55.816725 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:24:55.817639 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42011/ using document root <none> and password file <none>
I20260620 12:24:55.818198 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:55.818284 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:55.818511 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:55.820863 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "c24241cd6bcf4422629d0b112c39d04c"
server_key_iv: "61dddf8f70a33bcebadff4aa5480a284"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.821420 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "c24241cd6bcf4422629d0b112c39d04c"
server_key_iv: "61dddf8f70a33bcebadff4aa5480a284"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.825341 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:24:55.827951 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:55.829416 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 12:24:55.829550 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b"
format_stamp: "Formatted at 2026-06-20 12:24:55 on dist-test-slave-bwpp"
server_key: "c24241cd6bcf4422629d0b112c39d04c"
server_key_iv: "61dddf8f70a33bcebadff4aa5480a284"
server_key_version: "encryptionkey@0"
I20260620 12:24:55.829679 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:55.853212 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:55.856396 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:55.856629 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:55.857239 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:24:55.858078 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:55.858131 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:55.858173 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:55.858225 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:55.867276 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43693
I20260620 12:24:55.867295 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43693 every 8 connection(s)
I20260620 12:24:55.868223 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:24:55.874192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 1158
I20260620 12:24:55.874370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:24:55.874644 392 external_mini_cluster.cc:1428] Setting key e8686be741e56e0848b7213b0613fa66
Jun 20 12:24:55 dist-test-slave-bwpp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:55.883291 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:55.870144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55853 (local address 127.0.98.62:36061)
0620 12:24:55.870313 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:55.870318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:55.871213 (+ 895us) server_negotiation.cc:407] Connection header received
0620 12:24:55.872509 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:55.872515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:55.872573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:55.872661 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:55.873973 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.874586 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.875931 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.876345 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.876462 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:55.879121 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:55.879126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:55.879129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:55.879163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:55.880772 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:55.881326 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:55.881331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:55.881333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:55.881420 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:55.881920 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:55.881924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:55.881926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:55.882191 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:55.882305 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:55.882946 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 12:24:55.883085 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:24:55.884274 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:55.870552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36061 (local address 127.0.98.2:55853)
0620 12:24:55.871058 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:55.871102 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:55.872310 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:55.872876 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:55.872883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:55.873289 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:55.873812 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:55.873826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.874766 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.874772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:55.875776 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:55.875786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:55.876524 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:55.876528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:55.876704 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:55.877359 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:55.877377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:55.878945 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:55.880900 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:55.880907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:55.880925 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:55.881209 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:55.881546 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:55.881550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:55.881552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:55.881739 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:55.882311 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:55.882316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:55.882615 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 12:24:55.882850 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 12:24:55.883130 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":117,"threads_started":1}
I20260620 12:24:55.885528 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36061
I20260620 12:24:55.885845 1287 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:24:55.886489 1287 heartbeater.cc:511] Master 127.0.98.62:36061 requested a full tablet report, sending...
I20260620 12:24:55.887756 962 ts_manager.cc:194] Registered new tserver with Master: 38f8e8ac3dde4d03a8b9efcf0bccf19b (127.0.98.2:43693)
I20260620 12:24:55.888502 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:24:55.902424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:56.016649 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:56.016933 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:56.016979 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:56.021266 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:56.021343 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:56.021406 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:24:56.027019 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:56.028177 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:56.029192 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:56.038967 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:56.038964 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:56.039047 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:56.039206 1292 server_base.cc:1034] running on GCE node
I20260620 12:24:56.039700 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:56.040345 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:56.041525 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781958296041508 us; error 28 us; skew 500 ppm
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:56.048205 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:24:56.049274 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41389/ using document root <none> and password file <none>
I20260620 12:24:56.049872 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:56.049965 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:56.050213 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:56.052423 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192"
format_stamp: "Formatted at 2026-06-20 12:24:56 on dist-test-slave-bwpp"
server_key: "558710c0337dd2c5e52b8ecdc3594315"
server_key_iv: "23fc14a23fa9a5404467f51ed07c3b45"
server_key_version: "encryptionkey@0"
I20260620 12:24:56.053004 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192"
format_stamp: "Formatted at 2026-06-20 12:24:56 on dist-test-slave-bwpp"
server_key: "558710c0337dd2c5e52b8ecdc3594315"
server_key_iv: "23fc14a23fa9a5404467f51ed07c3b45"
server_key_version: "encryptionkey@0"
I20260620 12:24:56.056795 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260620 12:24:56.059391 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:56.060631 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:24:56.060766 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192"
format_stamp: "Formatted at 2026-06-20 12:24:56 on dist-test-slave-bwpp"
server_key: "558710c0337dd2c5e52b8ecdc3594315"
server_key_iv: "23fc14a23fa9a5404467f51ed07c3b45"
server_key_version: "encryptionkey@0"
I20260620 12:24:56.060874 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:56.076666 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:56.079835 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:56.080045 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:56.080648 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:24:56.081559 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:56.081622 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:56.081725 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:56.081769 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:56.091003 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40709
I20260620 12:24:56.091035 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40709 every 8 connection(s)
I20260620 12:24:56.091924 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:24:56.099054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 1292
I20260620 12:24:56.099198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:24:56.099475 392 external_mini_cluster.cc:1428] Setting key 7fad3aea1957f8efcf01a4e7e973693f
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:56.106324 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.093805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47459 (local address 127.0.98.62:36061)
0620 12:24:56.093970 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.093975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.094901 (+ 926us) server_negotiation.cc:407] Connection header received
0620 12:24:56.096215 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.096218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.096266 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.096344 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:56.097541 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.098044 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.099541 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.099836 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.099924 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.102815 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:56.102817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:56.102820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:56.102843 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:56.104377 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.104862 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.104865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.104866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.104937 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.105317 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.105320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.105322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.105503 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:56.105583 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.106097 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.106196 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:24:56.107366 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.094234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36061 (local address 127.0.98.3:47459)
0620 12:24:56.094754 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:56.094799 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:56.096015 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:56.096543 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:56.096550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:56.096927 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:56.097408 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.097422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.098178 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.098182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.099395 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.099406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.099999 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.100004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.100211 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.101000 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:56.101026 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:56.102602 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:56.104481 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:56.104486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:56.104498 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:56.104744 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:56.105033 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:56.105036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:56.105037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:56.105198 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:56.105617 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:56.105622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:56.105866 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 12:24:56.106030 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 12:24:56.106484 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260620 12:24:56.108450 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36061
I20260620 12:24:56.108680 1421 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:56.109149 1421 heartbeater.cc:511] Master 127.0.98.62:36061 requested a full tablet report, sending...
I20260620 12:24:56.110138 962 ts_manager.cc:194] Registered new tserver with Master: f1e83ce5eb7047e5ad7ff60d1e72f192 (127.0.98.3:40709)
I20260620 12:24:56.110761 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47459
I20260620 12:24:56.113332 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:24:56.123811 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.116529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.0.98.62:36061)
0620 12:24:56.116665 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.116671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.116877 (+ 206us) server_negotiation.cc:407] Connection header received
0620 12:24:56.117023 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.117026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.117070 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.117149 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:56.117830 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.118332 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.119286 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.119495 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.119599 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.121026 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:56.121030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:56.121037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:56.121059 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:56.122222 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.122627 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.122631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.122633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.122762 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.123046 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.123049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.123050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.123266 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:56.123375 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.123559 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.123690 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:24:56.128674 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:24:56.131744 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:24:56.147495 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.140416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43693 (local address 127.0.0.1:45028)
0620 12:24:56.141443 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:56.141459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:56.141570 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:56.142347 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:56.142350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:56.142366 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:56.142477 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.142496 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.143956 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.143959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.145458 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.145470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.146841 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.146845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.146974 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.147020 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 12:24:56.147147 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 12:24:56.147289 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260620 12:24:56.148201 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.140885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.0.98.2:43693)
0620 12:24:56.141428 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.141440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.141665 (+ 225us) server_negotiation.cc:407] Connection header received
0620 12:24:56.141750 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.141755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.141949 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.142179 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:56.142617 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.143812 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.145610 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.146468 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.146579 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.146777 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.147920 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.148032 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":156,"threads_started":1}
I20260620 12:24:56.150102 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.141188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.0.98.3:40709)
0620 12:24:56.141633 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.141646 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.142085 (+ 439us) server_negotiation.cc:407] Connection header received
0620 12:24:56.142187 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.142192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.142408 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.142557 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:56.143007 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.144103 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.146123 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.147288 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.147410 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.148064 (+ 654us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.149827 (+ 1763us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.149931 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":151,"threads_started":1}
I20260620 12:24:56.150578 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.141012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40709 (local address 127.0.0.1:39524)
0620 12:24:56.141791 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:56.141804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:56.141896 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:56.142756 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:56.142759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:56.142777 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:56.142864 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.142870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.144242 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.144245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.145923 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.145932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.148154 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.148157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.149542 (+ 1385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.149572 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:24:56.149637 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:24:56.149709 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":247,"spinlock_wait_cycles":768,"thread_start_us":76,"threads_started":1}
I20260620 12:24:56.151067 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.140260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38347 (local address 127.0.0.1:54192)
0620 12:24:56.140781 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:56.140843 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:56.140983 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:56.141820 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:56.141835 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:56.141857 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:56.142029 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.142036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.143979 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.143982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.149422 (+ 5440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.149433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.150627 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.150632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.150764 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.150789 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:24:56.150847 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 12:24:56.150929 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":4203,"spinlock_wait_cycles":3584,"thread_start_us":105,"threads_started":1}
I20260620 12:24:56.151682 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.140406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.0.98.1:38347)
0620 12:24:56.141258 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.141274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.141300 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:24:56.141379 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.141385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.141602 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.142182 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:56.142365 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.143818 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.149595 (+ 5777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.150474 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.151227 (+ 753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.151389 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.151440 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.151538 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":73,"threads_started":1}
I20260620 12:24:56.156385 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 1a16e5826baa4a35b8917a11dc4cf613 (DEFAULT_TABLE table=test-table [id=8fcfd33b7beb49e49bcfede85d6be41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:56.156574 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 1a16e5826baa4a35b8917a11dc4cf613 (DEFAULT_TABLE table=test-table [id=8fcfd33b7beb49e49bcfede85d6be41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:56.156703 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 1a16e5826baa4a35b8917a11dc4cf613 (DEFAULT_TABLE table=test-table [id=8fcfd33b7beb49e49bcfede85d6be41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:56.157585 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a16e5826baa4a35b8917a11dc4cf613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:56.157585 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a16e5826baa4a35b8917a11dc4cf613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:56.157914 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a16e5826baa4a35b8917a11dc4cf613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:56.166216 1439 tablet_bootstrap.cc:492] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192: Bootstrap starting.
I20260620 12:24:56.166409 1440 tablet_bootstrap.cc:492] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e: Bootstrap starting.
I20260620 12:24:56.166822 1438 tablet_bootstrap.cc:492] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b: Bootstrap starting.
I20260620 12:24:56.168620 1438 tablet_bootstrap.cc:654] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:56.168670 1439 tablet_bootstrap.cc:654] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:56.168670 1440 tablet_bootstrap.cc:654] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:56.169627 1440 log.cc:824] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:56.169675 1438 log.cc:824] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:56.169822 1439 log.cc:824] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:56.171907 1440 tablet_bootstrap.cc:492] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e: No bootstrap required, opened a new log
I20260620 12:24:56.172120 1440 ts_tablet_manager.cc:1397] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:24:56.172343 1438 tablet_bootstrap.cc:492] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b: No bootstrap required, opened a new log
I20260620 12:24:56.172415 1439 tablet_bootstrap.cc:492] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192: No bootstrap required, opened a new log
I20260620 12:24:56.172523 1438 ts_tablet_manager.cc:1397] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:24:56.172596 1439 ts_tablet_manager.cc:1397] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 12:24:56.175714 1439 raft_consensus.cc:348] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.176002 1439 raft_consensus.cc:374] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:56.176075 1439 raft_consensus.cc:729] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e83ce5eb7047e5ad7ff60d1e72f192, State: Initialized, Role: FOLLOWER
I20260620 12:24:56.176560 1439 consensus_queue.cc:260] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.176810 1438 raft_consensus.cc:348] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.176931 1440 raft_consensus.cc:348] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.177084 1438 raft_consensus.cc:374] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:56.177134 1438 raft_consensus.cc:729] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f8e8ac3dde4d03a8b9efcf0bccf19b, State: Initialized, Role: FOLLOWER
I20260620 12:24:56.177145 1440 raft_consensus.cc:374] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:56.177201 1440 raft_consensus.cc:729] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fb08479c274196a1585e6521fa997e, State: Initialized, Role: FOLLOWER
I20260620 12:24:56.177440 1421 heartbeater.cc:503] Master 127.0.98.62:36061 was elected leader, sending a full tablet report...
I20260620 12:24:56.177580 1440 consensus_queue.cc:260] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.178416 1153 heartbeater.cc:503] Master 127.0.98.62:36061 was elected leader, sending a full tablet report...
I20260620 12:24:56.178540 1440 ts_tablet_manager.cc:1428] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:24:56.177661 1439 ts_tablet_manager.cc:1428] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:24:56.180025 1438 consensus_queue.cc:260] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.180814 1287 heartbeater.cc:503] Master 127.0.98.62:36061 was elected leader, sending a full tablet report...
I20260620 12:24:56.181000 1438 ts_tablet_manager.cc:1428] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:24:56.183501 1446 raft_consensus.cc:482] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:24:56.183657 1446 raft_consensus.cc:504] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.184718 1446 leader_election.cc:288] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47fb08479c274196a1585e6521fa997e (127.0.98.1:38347), f1e83ce5eb7047e5ad7ff60d1e72f192 (127.0.98.3:40709)
I20260620 12:24:56.188539 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.184888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38347 (local address 127.0.98.2:48047)
0620 12:24:56.185000 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:56.185016 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:56.185101 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:56.185348 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:56.185350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:56.185366 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:56.185502 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.185510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.186338 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.186341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.187356 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.187366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.188205 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.188208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.188290 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.188310 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:24:56.188373 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:24:56.188423 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260620 12:24:56.188591 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.185013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48047 (local address 127.0.98.1:38347)
0620 12:24:56.185130 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.185134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.185148 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:24:56.185195 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.185198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.185246 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.185326 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:56.185619 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.186242 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.187483 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.188103 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.188203 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.188261 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.188404 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.188471 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 12:24:56.189013 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.184943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40709 (local address 127.0.98.2:43955)
0620 12:24:56.185350 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:56.185365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:56.185451 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:56.185683 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:56.185686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:56.185702 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:56.185805 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.185811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.186505 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.186507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.187562 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:56.187572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.188538 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.188542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:56.188634 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.188723 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 12:24:56.188823 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 12:24:56.188885 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":4,"thread_start_us":125,"threads_started":1}
I20260620 12:24:56.189030 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.185143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43955 (local address 127.0.98.3:40709)
0620 12:24:56.185265 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.185268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.185432 (+ 164us) server_negotiation.cc:407] Connection header received
0620 12:24:56.185531 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.185536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.185593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.185663 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:56.185897 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.186400 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.187691 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.188295 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.188395 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.188450 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.188834 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.188914 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:24:56.189105 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16e5826baa4a35b8917a11dc4cf613" candidate_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fb08479c274196a1585e6521fa997e" is_pre_election: true
I20260620 12:24:56.189389 1107 raft_consensus.cc:2454] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f8e8ac3dde4d03a8b9efcf0bccf19b in term 0.
I20260620 12:24:56.189642 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16e5826baa4a35b8917a11dc4cf613" candidate_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" is_pre_election: true
I20260620 12:24:56.189838 1176 leader_election.cc:302] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f8e8ac3dde4d03a8b9efcf0bccf19b, 47fb08479c274196a1585e6521fa997e; no voters:
I20260620 12:24:56.189881 1375 raft_consensus.cc:2454] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f8e8ac3dde4d03a8b9efcf0bccf19b in term 0.
I20260620 12:24:56.190099 1446 raft_consensus.cc:2781] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:24:56.190189 1446 raft_consensus.cc:482] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:24:56.190240 1446 raft_consensus.cc:3037] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:56.191334 1446 raft_consensus.cc:504] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.191682 1446 leader_election.cc:288] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [CANDIDATE]: Term 1 election: Requested vote from peers 47fb08479c274196a1585e6521fa997e (127.0.98.1:38347), f1e83ce5eb7047e5ad7ff60d1e72f192 (127.0.98.3:40709)
I20260620 12:24:56.192036 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16e5826baa4a35b8917a11dc4cf613" candidate_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192"
I20260620 12:24:56.192104 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16e5826baa4a35b8917a11dc4cf613" candidate_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fb08479c274196a1585e6521fa997e"
I20260620 12:24:56.192154 1375 raft_consensus.cc:3037] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:56.192301 1107 raft_consensus.cc:3037] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:56.193122 1375 raft_consensus.cc:2454] T 1a16e5826baa4a35b8917a11dc4cf613 P f1e83ce5eb7047e5ad7ff60d1e72f192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f8e8ac3dde4d03a8b9efcf0bccf19b in term 1.
I20260620 12:24:56.193392 1175 leader_election.cc:302] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f8e8ac3dde4d03a8b9efcf0bccf19b, f1e83ce5eb7047e5ad7ff60d1e72f192; no voters:
I20260620 12:24:56.193578 1107 raft_consensus.cc:2454] T 1a16e5826baa4a35b8917a11dc4cf613 P 47fb08479c274196a1585e6521fa997e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f8e8ac3dde4d03a8b9efcf0bccf19b in term 1.
I20260620 12:24:56.193634 1446 raft_consensus.cc:2781] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:56.193876 1446 raft_consensus.cc:686] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [term 1 LEADER]: Becoming Leader. State: Replica: 38f8e8ac3dde4d03a8b9efcf0bccf19b, State: Running, Role: LEADER
I20260620 12:24:56.194195 1446 consensus_queue.cc:237] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } }
I20260620 12:24:56.197734 961 catalog_manager.cc:5495] T 1a16e5826baa4a35b8917a11dc4cf613 P 38f8e8ac3dde4d03a8b9efcf0bccf19b reported cstate change: term changed from 0 to 1, leader changed from <none> to 38f8e8ac3dde4d03a8b9efcf0bccf19b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47fb08479c274196a1585e6521fa997e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38f8e8ac3dde4d03a8b9efcf0bccf19b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e83ce5eb7047e5ad7ff60d1e72f192" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40709 } health_report { overall_health: UNKNOWN } } }
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:24:56.221593 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.212382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.0.98.1:38347)
0620 12:24:56.212548 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.212552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.212758 (+ 206us) server_negotiation.cc:407] Connection header received
0620 12:24:56.212904 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.212908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.212972 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.213074 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:56.213766 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.214318 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.215226 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.215458 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.215541 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.217916 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:56.217930 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:56.217945 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:56.217971 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:56.219854 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.220286 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.220292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.220293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.220407 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.220621 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.220623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.220624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.220943 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:56.221102 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.221245 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.221353 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:24:56.249078 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.241073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.0.98.1:38347)
0620 12:24:56.241224 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.241229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.241348 (+ 119us) server_negotiation.cc:407] Connection header received
0620 12:24:56.241553 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.241556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.241606 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.241686 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:56.242347 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.242948 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.243770 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.244003 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.244107 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.246368 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:56.246372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:56.246374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:56.246400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:56.247706 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.248087 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.248090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.248092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.248159 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.248366 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.248368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.248369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.248532 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:56.248628 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.248862 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.248957 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:24:56.275593 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:56.266738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.0.98.1:38347)
0620 12:24:56.266901 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:56.266906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:56.267058 (+ 152us) server_negotiation.cc:407] Connection header received
0620 12:24:56.267239 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:56.267243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:56.267297 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:56.267381 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:56.268081 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.268593 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.269460 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:56.269693 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:56.269794 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:56.272274 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:56.272277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:56.272279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:56.272303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:56.273770 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.274155 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.274158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.274159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.274232 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:56.274446 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:56.274449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:56.274450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:56.274608 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:56.275053 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:56.275354 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 12:24:56.275471 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 12:24:56.276949 1067 tablet_service.cc:2908] Rejecting scan request for tablet 1a16e5826baa4a35b8917a11dc4cf613: Uninitialized: clean time has not yet been initialized
I20260620 12:24:56.279016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 1024
I20260620 12:24:56.284778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 1158
I20260620 12:24:56.290364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 1292
I20260620 12:24:56.295732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 930
2026-06-20T12:24:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3114 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[1462](info): set up 2 sockets
Jun 20 12:24:56 dist-test-slave-bwpp krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260620 12:24:57.342350 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:57 dist-test-slave-bwpp krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:57.462975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42461
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:57.573158 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:57.573506 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:57.573585 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:57.577837 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:57.577948 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:24:57.577982 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:57.578027 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:57.578063 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:57.578099 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:57.583534 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42461
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:57.584684 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:57.585777 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:24:57.594439 1475 server_base.cc:1034] running on GCE node
W20260620 12:24:57.594408 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:57.594455 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:57.594544 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:57.595151 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:57.596222 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:57.597409 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781958297597381 us; error 45 us; skew 500 ppm
Jun 20 12:24:57 dist-test-slave-bwpp krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:57.600282 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:57.601235 1475 webserver.cc:460] Webserver started at http://127.0.98.62:32823/ using document root <none> and password file <none>
I20260620 12:24:57.601799 1475 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:57.601903 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:57.602128 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:57.604223 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "df24dd3e196146759f850d6b4a6e1183"
format_stamp: "Formatted at 2026-06-20 12:24:57 on dist-test-slave-bwpp"
server_key: "49ae4e771e37195c98cd9e3e23e1c1af"
server_key_iv: "57ac1119b6d505c5bb913de8b7e13e3c"
server_key_version: "encryptionkey@0"
I20260620 12:24:57.604777 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "df24dd3e196146759f850d6b4a6e1183"
format_stamp: "Formatted at 2026-06-20 12:24:57 on dist-test-slave-bwpp"
server_key: "49ae4e771e37195c98cd9e3e23e1c1af"
server_key_iv: "57ac1119b6d505c5bb913de8b7e13e3c"
server_key_version: "encryptionkey@0"
I20260620 12:24:57.608480 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 12:24:57.610880 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:57.612135 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:24:57.612279 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "df24dd3e196146759f850d6b4a6e1183"
format_stamp: "Formatted at 2026-06-20 12:24:57 on dist-test-slave-bwpp"
server_key: "49ae4e771e37195c98cd9e3e23e1c1af"
server_key_iv: "57ac1119b6d505c5bb913de8b7e13e3c"
server_key_version: "encryptionkey@0"
I20260620 12:24:57.612393 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:57.638468 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:57.641953 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:57.642181 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:57.649839 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42461
I20260620 12:24:57.649893 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42461 every 8 connection(s)
I20260620 12:24:57.650899 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:57.654048 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:57.659369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 1475
I20260620 12:24:57.659488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:57.659739 392 external_mini_cluster.cc:1428] Setting key 6384645d341d3376b2e7b41409cbeb85
I20260620 12:24:57.660192 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183: Bootstrap starting.
I20260620 12:24:57.662634 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:57.663662 1553 log.cc:824] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:57.665787 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183: No bootstrap required, opened a new log
Jun 20 12:24:57 dist-test-slave-bwpp krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:57.670061 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df24dd3e196146759f850d6b4a6e1183" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } }
I20260620 12:24:57.670291 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:57.670326 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df24dd3e196146759f850d6b4a6e1183, State: Initialized, Role: FOLLOWER
I20260620 12:24:57.670850 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df24dd3e196146759f850d6b4a6e1183" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } }
I20260620 12:24:57.671003 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:24:57.671085 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:24:57.671196 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:57.672096 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df24dd3e196146759f850d6b4a6e1183" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } }
I20260620 12:24:57.672432 1553 leader_election.cc:302] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df24dd3e196146759f850d6b4a6e1183; no voters:
I20260620 12:24:57.672806 1553 leader_election.cc:288] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:24:57.672881 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:57.673059 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [term 1 LEADER]: Becoming Leader. State: Replica: df24dd3e196146759f850d6b4a6e1183, State: Running, Role: LEADER
I20260620 12:24:57.673430 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df24dd3e196146759f850d6b4a6e1183" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } }
I20260620 12:24:57.674142 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:57.675208 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:57.661190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.0.98.62:42461)
0620 12:24:57.661610 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:57.661631 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:57.661668 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:24:57.662311 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:57.662329 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:57.662691 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:57.663045 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:57.663932 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.664920 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.666047 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.666512 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.666631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:57.669444 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:57.669450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:57.669465 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:57.669500 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:57.671500 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:57.671945 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:57.671951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:57.671957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:57.672081 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:57.672381 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:57.672384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:57.672386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:57.672843 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:57.673758 (+ 915us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:57.673879 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 12:24:57.674157 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":158,"threads_started":1}
I20260620 12:24:57.675170 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df24dd3e196146759f850d6b4a6e1183. Latest consensus state: current_term: 1 leader_uuid: "df24dd3e196146759f850d6b4a6e1183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df24dd3e196146759f850d6b4a6e1183" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } }
I20260620 12:24:57.675361 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df24dd3e196146759f850d6b4a6e1183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df24dd3e196146759f850d6b4a6e1183" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } }
I20260620 12:24:57.675449 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:57.675863 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:57.675452 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:57.679096 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:57.686607 1567 catalog_manager.cc:1347] Generated new cluster ID: e2d801959ec34506a12058733b653eb2
I20260620 12:24:57.686733 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:57.704237 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:57.705258 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:57.719551 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183: Generated new TSK 0
I20260620 12:24:57.720310 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:24:57.753517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42461
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:57.868201 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:57.868482 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:57.868530 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:57.872694 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:57.872756 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:57.872797 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:24:57.872862 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:24:57.878434 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:57.879592 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:57.880647 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:57.890560 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:57.890573 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:57.890573 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:57.890777 1579 server_base.cc:1034] running on GCE node
I20260620 12:24:57.891368 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:57.892011 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:57.893167 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781958297893132 us; error 48 us; skew 500 ppm
Jun 20 12:24:57 dist-test-slave-bwpp krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:57.896008 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:24:57.896905 1579 webserver.cc:460] Webserver started at http://127.0.98.1:46417/ using document root <none> and password file <none>
I20260620 12:24:57.897461 1579 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:57.897560 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:57.897782 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:57.899868 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "c98c7d9253ca484c8f9036a7af3c784c"
format_stamp: "Formatted at 2026-06-20 12:24:57 on dist-test-slave-bwpp"
server_key: "4919d95353fb9a60973c6500a79d45db"
server_key_iv: "e8253fd8f1f0d2d81d8ab4571dfbbf4f"
server_key_version: "encryptionkey@0"
I20260620 12:24:57.900429 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c98c7d9253ca484c8f9036a7af3c784c"
format_stamp: "Formatted at 2026-06-20 12:24:57 on dist-test-slave-bwpp"
server_key: "4919d95353fb9a60973c6500a79d45db"
server_key_iv: "e8253fd8f1f0d2d81d8ab4571dfbbf4f"
server_key_version: "encryptionkey@0"
I20260620 12:24:57.904148 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:24:57.906594 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:57.907833 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:57.907974 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "c98c7d9253ca484c8f9036a7af3c784c"
format_stamp: "Formatted at 2026-06-20 12:24:57 on dist-test-slave-bwpp"
server_key: "4919d95353fb9a60973c6500a79d45db"
server_key_iv: "e8253fd8f1f0d2d81d8ab4571dfbbf4f"
server_key_version: "encryptionkey@0"
I20260620 12:24:57.908087 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:57.921911 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:57.925379 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:57.925598 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:57.927201 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:57.927271 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:57.927337 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:57.927376 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:57.938267 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37421
Jun 20 12:24:57 dist-test-slave-bwpp krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:57.938614 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37421 every 8 connection(s)
I20260620 12:24:57.939343 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:24:57.940521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 1579
I20260620 12:24:57.940623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:24:57.940835 392 external_mini_cluster.cc:1428] Setting key 6333f37979d1b04abd164f2a8db76ff1
I20260620 12:24:57.944010 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:57.927981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35001 (local address 127.0.98.62:42461)
0620 12:24:57.928134 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:57.928139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:57.929134 (+ 995us) server_negotiation.cc:407] Connection header received
0620 12:24:57.930486 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:57.930491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:57.930560 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:57.930629 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:57.932638 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.933411 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.935304 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.935554 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.935646 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:57.939097 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:57.939100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:57.939102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:57.939129 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:57.941721 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:57.942332 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:57.942336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:57.942338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:57.942436 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:57.942833 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:57.942839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:57.942844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:57.943110 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:57.943216 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:57.943682 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 12:24:57.943847 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:24:57.945114 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:57.928440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.98.1:35001)
0620 12:24:57.928935 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:57.929012 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:57.930201 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:57.931149 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:57.931162 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:57.931687 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:57.932458 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:57.932486 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.933570 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.933574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:57.935197 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:57.935207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.935720 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.935727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:57.935981 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:57.936845 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:57.936871 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:57.938914 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:57.941830 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:57.941836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:57.941851 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:57.942220 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:57.942528 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:57.942530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:57.942532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:57.942706 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:57.943207 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:57.943212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:57.943441 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 12:24:57.943592 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 12:24:57.943971 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260620 12:24:57.946502 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42461
I20260620 12:24:57.946698 1710 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:57.947347 1710 heartbeater.cc:511] Master 127.0.98.62:42461 requested a full tablet report, sending...
I20260620 12:24:57.949107 1505 ts_manager.cc:194] Registered new tserver with Master: c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421)
I20260620 12:24:57.950222 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35001
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:24:57.958441 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:57.948705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36895 (local address 127.0.98.62:42461)
0620 12:24:57.948831 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:57.948835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:57.951217 (+ 2382us) server_negotiation.cc:407] Connection header received
0620 12:24:57.951320 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:57.951322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:57.951359 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:57.951424 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:57.951975 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.952466 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.953549 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.953753 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.953829 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:57.955130 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:57.955136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:57.955141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:57.955172 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:57.956723 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:57.957262 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:57.957269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:57.957272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:57.957357 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:57.957659 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:57.957661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:57.957663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:57.957829 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:57.957903 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:57.958224 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 12:24:57.958320 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:24:57.958737 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:57.948638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.98.1:36895)
0620 12:24:57.951122 (+ 2484us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:57.951139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:57.951240 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:57.951466 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:57.951469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:57.951717 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:57.951869 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:57.951878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.952604 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.952607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:57.953419 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:57.953431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:57.953873 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:57.953879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:57.954007 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:57.954519 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:57.954538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:57.954994 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:57.956840 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:57.956846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:57.956852 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:57.957147 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:57.957448 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:57.957454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:57.957458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:57.957564 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:57.957926 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:57.957932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:57.958054 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 12:24:57.958184 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 12:24:57.958378 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2383,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:24:57.970256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42461
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:58.082695 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:58.082966 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:58.083016 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:58.087373 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:58.087431 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:58.087500 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:24:58.087575 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:24:58.092868 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:58.093933 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:58.095028 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:58.104719 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:58.104719 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:58.104755 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:58.105010 1719 server_base.cc:1034] running on GCE node
I20260620 12:24:58.105432 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:58.106076 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:58.107232 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781958298107203 us; error 41 us; skew 500 ppm
Jun 20 12:24:58 dist-test-slave-bwpp krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:58.109953 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:24:58.110939 1719 webserver.cc:460] Webserver started at http://127.0.98.2:40597/ using document root <none> and password file <none>
I20260620 12:24:58.111546 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:58.111613 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:58.111871 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:58.114097 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "d861de46115748d5b716b4f4774f189a"
format_stamp: "Formatted at 2026-06-20 12:24:58 on dist-test-slave-bwpp"
server_key: "d8caaaccff1c92af1111083b1832f251"
server_key_iv: "a5769e1a85840dab230e910a6193fb72"
server_key_version: "encryptionkey@0"
I20260620 12:24:58.114789 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d861de46115748d5b716b4f4774f189a"
format_stamp: "Formatted at 2026-06-20 12:24:58 on dist-test-slave-bwpp"
server_key: "d8caaaccff1c92af1111083b1832f251"
server_key_iv: "a5769e1a85840dab230e910a6193fb72"
server_key_version: "encryptionkey@0"
I20260620 12:24:58.119009 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:24:58.121522 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:58.122802 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:58.122928 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "d861de46115748d5b716b4f4774f189a"
format_stamp: "Formatted at 2026-06-20 12:24:58 on dist-test-slave-bwpp"
server_key: "d8caaaccff1c92af1111083b1832f251"
server_key_iv: "a5769e1a85840dab230e910a6193fb72"
server_key_version: "encryptionkey@0"
I20260620 12:24:58.123046 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:58.132550 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:58.135499 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:58.135690 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:58.137156 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:58.137233 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:58.137312 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:58.137367 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:58.148479 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45721
I20260620 12:24:58.148772 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45721 every 8 connection(s)
Jun 20 12:24:58 dist-test-slave-bwpp krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:58.149700 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:24:58.153661 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.138060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53027 (local address 127.0.98.62:42461)
0620 12:24:58.138178 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.138182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.139253 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 12:24:58.140194 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.140198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.140262 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.140362 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:58.142154 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.143132 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.145373 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.145780 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.145912 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.150012 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:58.150017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:58.150021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:58.150055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:58.151666 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.152166 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.152170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.152172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.152259 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.152589 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.152594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.152596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.152783 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:58.152883 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.153371 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.153494 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:24:58.154450 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.138626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.98.2:53027)
0620 12:24:58.139104 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.139144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.139960 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.140712 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.140725 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.141245 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:58.141958 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.141981 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.143303 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.143309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.144953 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.144966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.146002 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.146010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.146361 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.147154 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:58.147184 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:58.149811 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:58.151783 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.151789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.151803 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.152062 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.152349 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.152353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.152355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.152499 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.152891 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:58.152900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:58.153139 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.153315 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.153635 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260620 12:24:58.155630 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42461
I20260620 12:24:58.155771 1850 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:58.156405 1850 heartbeater.cc:511] Master 127.0.98.62:42461 requested a full tablet report, sending...
I20260620 12:24:58.156541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 1719
I20260620 12:24:58.156677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:24:58.156924 392 external_mini_cluster.cc:1428] Setting key f2e080e6d536b8853b3b22113218d87b
I20260620 12:24:58.157611 1507 ts_manager.cc:194] Registered new tserver with Master: d861de46115748d5b716b4f4774f189a (127.0.98.2:45721)
I20260620 12:24:58.158268 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53027
I20260620 12:24:58.165980 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.158200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.98.2:43937)
0620 12:24:58.158589 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.158604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.158725 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.158972 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.158974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.159165 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:58.159258 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.159263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.160240 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.160246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.161221 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.161231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.161671 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.161675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.161765 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.162306 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:58.162318 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:58.162722 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:58.164334 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.164339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.164341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.164616 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.164887 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.164890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.164891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.164979 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.165390 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:58.165393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:58.165497 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.165629 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.165785 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260620 12:24:58.165973 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.158592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43937 (local address 127.0.98.62:42461)
0620 12:24:58.158724 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.158734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.158747 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:24:58.158822 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.158826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.158863 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.158955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:58.159372 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.160083 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.161333 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.161563 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.161668 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.162865 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:58.162868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:58.162870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:58.162891 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:58.164225 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.164702 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.164707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.164708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.164789 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.165074 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.165077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.165079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.165293 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:58.165401 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.165696 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.165811 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:24:58.185528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42461
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:58.298476 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:58.298825 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:58.298910 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:58.305171 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:58.305258 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:58.305327 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:24:58.305397 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:24:58.310798 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:58.311976 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:58.313052 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:58.323032 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:58.323032 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:58.323133 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:58.323158 1859 server_base.cc:1034] running on GCE node
I20260620 12:24:58.323765 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:58.324405 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:58.325615 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781958298325601 us; error 37 us; skew 500 ppm
Jun 20 12:24:58 dist-test-slave-bwpp krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:58.328640 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:24:58.329514 1859 webserver.cc:460] Webserver started at http://127.0.98.3:38069/ using document root <none> and password file <none>
I20260620 12:24:58.330077 1859 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:58.330164 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:58.330387 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:58.332437 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "b45f21120b674406981e0b4e17c66d51"
format_stamp: "Formatted at 2026-06-20 12:24:58 on dist-test-slave-bwpp"
server_key: "ed9187f7ecf6440adb2b75a765aa38ac"
server_key_iv: "360400815530037a0903bf656960526a"
server_key_version: "encryptionkey@0"
I20260620 12:24:58.332998 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b45f21120b674406981e0b4e17c66d51"
format_stamp: "Formatted at 2026-06-20 12:24:58 on dist-test-slave-bwpp"
server_key: "ed9187f7ecf6440adb2b75a765aa38ac"
server_key_iv: "360400815530037a0903bf656960526a"
server_key_version: "encryptionkey@0"
I20260620 12:24:58.336743 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:24:58.339110 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:58.340287 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:58.340445 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "b45f21120b674406981e0b4e17c66d51"
format_stamp: "Formatted at 2026-06-20 12:24:58 on dist-test-slave-bwpp"
server_key: "ed9187f7ecf6440adb2b75a765aa38ac"
server_key_iv: "360400815530037a0903bf656960526a"
server_key_version: "encryptionkey@0"
I20260620 12:24:58.340543 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:58.360786 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:58.363708 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:58.363901 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:58.365345 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:24:58.365412 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:58.365479 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:24:58.365523 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:24:58 dist-test-slave-bwpp krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:58.376528 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46373
I20260620 12:24:58.376613 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46373 every 8 connection(s)
I20260620 12:24:58.377707 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:24:58.379510 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.366134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54303 (local address 127.0.98.62:42461)
0620 12:24:58.366251 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.366254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.367314 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 12:24:58.368330 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.368333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.368392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.368456 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:58.369942 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.370901 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.372223 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.372624 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.372749 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.375774 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:58.375777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:58.375780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:58.375815 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:58.377482 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.378034 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.378038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.378040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.378137 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.378535 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.378538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.378540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.378798 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:58.378884 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.379285 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.379379 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:24:58.380304 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.366714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.98.3:54303)
0620 12:24:58.367177 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.367219 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.368085 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.368819 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.368829 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.369224 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:58.369702 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.369716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.371041 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.371044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.372120 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.372129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.372793 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.372799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.373015 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.373600 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:58.373619 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:58.375596 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:58.377610 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.377615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.377625 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.377937 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.378248 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.378251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.378254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.378434 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.378880 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:58.378884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:58.379091 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.379240 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.379469 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
I20260620 12:24:58.381580 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42461
I20260620 12:24:58.381726 1990 heartbeater.cc:464] Registering TS with master...
I20260620 12:24:58.381912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 1859
I20260620 12:24:58.382036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:24:58.382280 1990 heartbeater.cc:511] Master 127.0.98.62:42461 requested a full tablet report, sending...
I20260620 12:24:58.382366 392 external_mini_cluster.cc:1428] Setting key c7bbadddc6dc6e20f1015f8d4f801286
I20260620 12:24:58.383405 1505 ts_manager.cc:194] Registered new tserver with Master: b45f21120b674406981e0b4e17c66d51 (127.0.98.3:46373)
I20260620 12:24:58.384137 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54303
I20260620 12:24:58.384404 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:24:58.391829 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.383805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49579 (local address 127.0.98.62:42461)
0620 12:24:58.384444 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.384448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.384461 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:24:58.384501 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.384504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.384542 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.384620 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:58.385040 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.385536 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.386728 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.387041 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.387149 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.388450 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:58.388456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:58.388460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:58.388489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:58.390185 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.390669 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.390674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.390676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.390758 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.391057 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.391059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.391060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.391214 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:58.391288 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.391597 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.391703 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575}
I20260620 12:24:58.392107 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.383832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.98.3:49579)
0620 12:24:58.384101 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.384116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.384205 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.384637 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.384639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.384832 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:58.384921 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.384926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.385654 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.385657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.386567 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.386580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.387168 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.387175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.387301 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.387796 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:58.387816 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:58.388289 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:58.390305 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.390308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.390310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.390537 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.390847 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.390852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.390855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.390959 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.391301 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:58.391305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:58.391408 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.391535 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.391690 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
Jun 20 12:24:58 dist-test-slave-bwpp krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:24:58 dist-test-slave-bwpp krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:58.421048 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.411423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.0.98.62:42461)
0620 12:24:58.411570 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.411575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.411594 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:24:58.411640 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.411643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.411685 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.411768 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:58.412904 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.413471 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.414851 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.415063 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.415144 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.418137 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:58.418140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:58.418142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:58.418166 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:58.419523 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.419919 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.419921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.419923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.419995 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.420222 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.420224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.420225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.420448 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:58.420562 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.420782 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.420873 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:24:58.424266 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:24:58.427637 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:24:58.441964 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.436167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46373 (local address 127.0.0.1:51370)
0620 12:24:58.436549 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.436609 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.436735 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.437915 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.437921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.437942 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:58.438083 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.438090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.439264 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.439269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.440431 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.440441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.441414 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.441418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.441528 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.441564 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.441684 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.441803 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260620 12:24:58.441985 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.436908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.98.3:46373)
0620 12:24:58.437229 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.437242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.437267 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:24:58.437343 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.437348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.437596 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.437754 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:58.438206 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.439157 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.440569 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.441301 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.441416 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.441593 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.441712 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.441803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":145,"threads_started":1}
I20260620 12:24:58.442767 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.436958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37421 (local address 127.0.0.1:58520)
0620 12:24:58.437559 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.437572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.437713 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.438285 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.438287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.438310 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:58.438376 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.438381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.439801 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.439804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.440961 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.440970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.442236 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.442240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.442401 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.442424 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.442501 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.442572 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":66,"spinlock_wait_cycles":2944,"thread_start_us":85,"threads_started":1}
I20260620 12:24:58.442803 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.437078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.0.98.1:37421)
0620 12:24:58.437366 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.437379 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.437647 (+ 268us) server_negotiation.cc:407] Connection header received
0620 12:24:58.437864 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.437886 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.438061 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.438158 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:58.438482 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.439679 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.441081 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.441892 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.442024 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.442173 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.442557 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.442644 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":52,"threads_started":1}
I20260620 12:24:58.443128 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.436559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.0.98.2:45721)
0620 12:24:58.436954 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.436966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.437590 (+ 624us) server_negotiation.cc:407] Connection header received
0620 12:24:58.437803 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.437808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.437998 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.438117 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:58.438555 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.439691 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.441136 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.441899 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.442024 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.442173 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.442835 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.442962 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":97,"threads_started":1}
I20260620 12:24:58.443429 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.436336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45721 (local address 127.0.0.1:38720)
0620 12:24:58.437493 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.437506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.437660 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.438286 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.438289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.438317 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:58.438397 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.438402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.439842 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.439845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.440967 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.440975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.442236 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.442240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.442580 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.442600 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.442687 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.442751 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":377,"spinlock_wait_cycles":2688,"thread_start_us":116,"threads_started":1}
I20260620 12:24:58.445458 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 3a72cb47b25940a7b658b159f50d1da5 (DEFAULT_TABLE table=test-table [id=771df14ecaf641888b8617e23d3a6cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:58.445504 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 3a72cb47b25940a7b658b159f50d1da5 (DEFAULT_TABLE table=test-table [id=771df14ecaf641888b8617e23d3a6cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:58.446048 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 3a72cb47b25940a7b658b159f50d1da5 (DEFAULT_TABLE table=test-table [id=771df14ecaf641888b8617e23d3a6cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:58.446524 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a72cb47b25940a7b658b159f50d1da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.446524 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a72cb47b25940a7b658b159f50d1da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.446945 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a72cb47b25940a7b658b159f50d1da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.452520 2013 tablet_bootstrap.cc:492] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Bootstrap starting.
I20260620 12:24:58.454032 2014 tablet_bootstrap.cc:492] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Bootstrap starting.
I20260620 12:24:58.454425 2015 tablet_bootstrap.cc:492] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Bootstrap starting.
I20260620 12:24:58.455699 2013 tablet_bootstrap.cc:654] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.456588 2015 tablet_bootstrap.cc:654] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.456598 2013 log.cc:824] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:58.456708 2014 tablet_bootstrap.cc:654] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.457299 2015 log.cc:824] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:58.457490 2014 log.cc:824] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:58.458729 2013 tablet_bootstrap.cc:492] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: No bootstrap required, opened a new log
I20260620 12:24:58.458926 2013 ts_tablet_manager.cc:1397] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:24:58.459160 2015 tablet_bootstrap.cc:492] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: No bootstrap required, opened a new log
I20260620 12:24:58.459333 2015 ts_tablet_manager.cc:1397] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:24:58.459380 2014 tablet_bootstrap.cc:492] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: No bootstrap required, opened a new log
I20260620 12:24:58.459532 2014 ts_tablet_manager.cc:1397] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:24:58.462491 2015 raft_consensus.cc:348] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.462750 2015 raft_consensus.cc:374] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.462790 2015 raft_consensus.cc:729] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d861de46115748d5b716b4f4774f189a, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.462782 2013 raft_consensus.cc:348] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.463038 2013 raft_consensus.cc:374] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.463097 2013 raft_consensus.cc:729] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c98c7d9253ca484c8f9036a7af3c784c, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.463169 2015 consensus_queue.cc:260] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.463158 2014 raft_consensus.cc:348] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.463363 2014 raft_consensus.cc:374] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.463410 2014 raft_consensus.cc:729] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45f21120b674406981e0b4e17c66d51, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.463521 2013 consensus_queue.cc:260] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.463790 1850 heartbeater.cc:503] Master 127.0.98.62:42461 was elected leader, sending a full tablet report...
I20260620 12:24:58.463781 2014 consensus_queue.cc:260] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.464493 2013 ts_tablet_manager.cc:1428] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:24:58.464522 2015 ts_tablet_manager.cc:1428] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:24:58.464588 2014 ts_tablet_manager.cc:1428] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:24:58.464814 1710 heartbeater.cc:503] Master 127.0.98.62:42461 was elected leader, sending a full tablet report...
I20260620 12:24:58.464815 1990 heartbeater.cc:503] Master 127.0.98.62:42461 was elected leader, sending a full tablet report...
I20260620 12:24:58.483119 2021 raft_consensus.cc:482] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:24:58.483311 2021 raft_consensus.cc:504] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.484372 2021 leader_election.cc:288] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d861de46115748d5b716b4f4774f189a (127.0.98.2:45721), c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421)
I20260620 12:24:58.488392 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.484594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45721 (local address 127.0.98.3:54639)
0620 12:24:58.484730 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.484744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.484843 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.485158 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.485164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.485185 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:58.485323 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.485331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.486240 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.486247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.487203 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.487212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.487955 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.487959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.488080 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.488098 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.488189 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.488262 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 12:24:58.488391 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.484766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54639 (local address 127.0.98.2:45721)
0620 12:24:58.484916 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.484923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.484938 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:24:58.484986 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.484989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.485051 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.485126 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:58.485448 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.486138 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.487328 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.487841 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.487955 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.488023 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.488214 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.488277 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:24:58.488891 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.484860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37421 (local address 127.0.98.3:38517)
0620 12:24:58.485188 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.485201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.485298 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.485527 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.485529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.485546 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:58.485661 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.485667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.486610 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.486613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.487821 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.487831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.488458 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.488461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.488554 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.488580 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.488690 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.488758 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260620 12:24:58.488907 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.485001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38517 (local address 127.0.98.1:37421)
0620 12:24:58.485141 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.485146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.485273 (+ 127us) server_negotiation.cc:407] Connection header received
0620 12:24:58.485369 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.485373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.485431 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.485500 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:58.485770 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.486400 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.487943 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.488327 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.488451 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.488507 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.488715 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.488784 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:24:58.489127 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a72cb47b25940a7b658b159f50d1da5" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d861de46115748d5b716b4f4774f189a" is_pre_election: true
I20260620 12:24:58.489418 1804 raft_consensus.cc:2454] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 0.
I20260620 12:24:58.489573 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a72cb47b25940a7b658b159f50d1da5" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c98c7d9253ca484c8f9036a7af3c784c" is_pre_election: true
I20260620 12:24:58.489831 1664 raft_consensus.cc:2454] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 0.
I20260620 12:24:58.489872 1877 leader_election.cc:302] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45f21120b674406981e0b4e17c66d51, d861de46115748d5b716b4f4774f189a; no voters:
I20260620 12:24:58.490108 2021 raft_consensus.cc:2781] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:24:58.490192 2021 raft_consensus.cc:482] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:24:58.490247 2021 raft_consensus.cc:3037] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:58.491380 2021 raft_consensus.cc:504] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.491701 2021 leader_election.cc:288] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 election: Requested vote from peers d861de46115748d5b716b4f4774f189a (127.0.98.2:45721), c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421)
I20260620 12:24:58.491994 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a72cb47b25940a7b658b159f50d1da5" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d861de46115748d5b716b4f4774f189a"
I20260620 12:24:58.492097 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a72cb47b25940a7b658b159f50d1da5" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c98c7d9253ca484c8f9036a7af3c784c"
I20260620 12:24:58.492116 1804 raft_consensus.cc:3037] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:58.492229 1664 raft_consensus.cc:3037] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:58.493131 1804 raft_consensus.cc:2454] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 1.
I20260620 12:24:58.493281 1664 raft_consensus.cc:2454] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 1.
I20260620 12:24:58.493409 1877 leader_election.cc:302] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45f21120b674406981e0b4e17c66d51, d861de46115748d5b716b4f4774f189a; no voters:
I20260620 12:24:58.493686 2021 raft_consensus.cc:2781] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:58.493851 2021 raft_consensus.cc:686] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 1 LEADER]: Becoming Leader. State: Replica: b45f21120b674406981e0b4e17c66d51, State: Running, Role: LEADER
I20260620 12:24:58.494194 2021 consensus_queue.cc:237] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } }
I20260620 12:24:58.497253 1505 catalog_manager.cc:5495] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b45f21120b674406981e0b4e17c66d51 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } health_report { overall_health: UNKNOWN } } }
I20260620 12:24:58.528163 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.524449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.0.98.3:46373)
0620 12:24:58.524575 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.524578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.524592 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:24:58.524716 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.524718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.524781 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.524869 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:58.525098 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.525633 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.526582 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.526874 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.526969 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.527284 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:58.527810 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:58.527909 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.528003 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.528054 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:24:58.533753 1804 raft_consensus.cc:1261] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Refusing update from remote peer b45f21120b674406981e0b4e17c66d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:58.533753 1664 raft_consensus.cc:1261] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Refusing update from remote peer b45f21120b674406981e0b4e17c66d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:58.534436 2023 consensus_queue.cc:1035] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:58.534642 2021 consensus_queue.cc:1035] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:58.541647 2026 mvcc.cc:205] Tried to move back new op lower bound from 7298901190787313664 to 7298901190635290624. Current Snapshot: MvccSnapshot[applied={T|T < 7298901190787313664}]
I20260620 12:24:58.544582 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298901190787313664 to 7298901190635290624. Current Snapshot: MvccSnapshot[applied={T|T < 7298901190787313664}]
I20260620 12:24:58.553083 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:24:58.553299 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:24:58.555603 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183: Sending DeleteTablet for 3 replicas of tablet 3a72cb47b25940a7b658b159f50d1da5
I20260620 12:24:58.556311 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a72cb47b25940a7b658b159f50d1da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:24:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51370
I20260620 12:24:58.556398 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a72cb47b25940a7b658b159f50d1da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:24:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38720
I20260620 12:24:58.556855 2035 tablet_replica.cc:330] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: stopping tablet replica
I20260620 12:24:58.556842 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a72cb47b25940a7b658b159f50d1da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:24:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260620 12:24:58.557197 2035 raft_consensus.cc:2229] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:24:58.557335 2037 tablet_replica.cc:330] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: stopping tablet replica
I20260620 12:24:58.557528 2035 raft_consensus.cc:2258] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:24:58.557709 2037 raft_consensus.cc:2229] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:24:58.557875 2036 tablet_replica.cc:330] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: stopping tablet replica
I20260620 12:24:58.558055 2037 raft_consensus.cc:2258] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:24:58.558182 2036 raft_consensus.cc:2229] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:24:58.558239 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:24:58.558597 2036 raft_consensus.cc:2258] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 12:24:58.558641 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:24:58.559865 2036 ts_tablet_manager.cc:1905] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:24:58.559867 2037 ts_tablet_manager.cc:1905] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:24:58.562567 2037 ts_tablet_manager.cc:1918] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:24:58.562587 2036 ts_tablet_manager.cc:1918] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:24:58.562695 2036 log.cc:1196] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/3a72cb47b25940a7b658b159f50d1da5
I20260620 12:24:58.562824 2037 log.cc:1196] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/3a72cb47b25940a7b658b159f50d1da5
I20260620 12:24:58.563005 2036 ts_tablet_manager.cc:1939] T 3a72cb47b25940a7b658b159f50d1da5 P b45f21120b674406981e0b4e17c66d51: Deleting consensus metadata
I20260620 12:24:58.563279 2037 ts_tablet_manager.cc:1939] T 3a72cb47b25940a7b658b159f50d1da5 P c98c7d9253ca484c8f9036a7af3c784c: Deleting consensus metadata
I20260620 12:24:58.563941 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 31c1a37bb1254fdfafe46050dbb80a23 (DEFAULT_TABLE table=test-table [id=0e566b21513e4737a29bc751289a76e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:58.564111 1494 catalog_manager.cc:4841] TS b45f21120b674406981e0b4e17c66d51 (127.0.98.3:46373): tablet 3a72cb47b25940a7b658b159f50d1da5 (table test-table [id=771df14ecaf641888b8617e23d3a6cad]) successfully deleted
I20260620 12:24:58.564203 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c1a37bb1254fdfafe46050dbb80a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.565834 2014 tablet_bootstrap.cc:492] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Bootstrap starting.
I20260620 12:24:58.566057 1493 catalog_manager.cc:4841] TS c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421): tablet 3a72cb47b25940a7b658b159f50d1da5 (table test-table [id=771df14ecaf641888b8617e23d3a6cad]) successfully deleted
I20260620 12:24:58.566345 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 31c1a37bb1254fdfafe46050dbb80a23 (DEFAULT_TABLE table=test-table [id=0e566b21513e4737a29bc751289a76e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:58.566574 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c1a37bb1254fdfafe46050dbb80a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.566561 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 31c1a37bb1254fdfafe46050dbb80a23 (DEFAULT_TABLE table=test-table [id=0e566b21513e4737a29bc751289a76e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:24:58.566833 2014 tablet_bootstrap.cc:654] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.566839 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c1a37bb1254fdfafe46050dbb80a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.568050 2014 tablet_bootstrap.cc:492] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: No bootstrap required, opened a new log
I20260620 12:24:58.568136 2014 ts_tablet_manager.cc:1397] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:58.568603 2014 raft_consensus.cc:348] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.568698 2014 raft_consensus.cc:374] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.568728 2014 raft_consensus.cc:729] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45f21120b674406981e0b4e17c66d51, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.568847 2014 consensus_queue.cc:260] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.569121 2014 ts_tablet_manager.cc:1428] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:24:58.569248 2015 tablet_bootstrap.cc:492] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: Bootstrap starting.
I20260620 12:24:58.569355 2013 tablet_bootstrap.cc:492] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: Bootstrap starting.
I20260620 12:24:58.570222 2015 tablet_bootstrap.cc:654] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.570222 2013 tablet_bootstrap.cc:654] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.572448 2035 ts_tablet_manager.cc:1905] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:24:58.574572 2035 ts_tablet_manager.cc:1918] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:24:58.574702 2035 log.cc:1196] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/3a72cb47b25940a7b658b159f50d1da5
I20260620 12:24:58.575033 2035 ts_tablet_manager.cc:1939] T 3a72cb47b25940a7b658b159f50d1da5 P d861de46115748d5b716b4f4774f189a: Deleting consensus metadata
I20260620 12:24:58.576153 1493 catalog_manager.cc:4841] TS d861de46115748d5b716b4f4774f189a (127.0.98.2:45721): tablet 3a72cb47b25940a7b658b159f50d1da5 (table test-table [id=771df14ecaf641888b8617e23d3a6cad]) successfully deleted
I20260620 12:24:58.576534 2015 tablet_bootstrap.cc:492] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: No bootstrap required, opened a new log
I20260620 12:24:58.576628 2015 ts_tablet_manager.cc:1397] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 12:24:58.577054 2015 raft_consensus.cc:348] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.577143 2015 raft_consensus.cc:374] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.577164 2015 raft_consensus.cc:729] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d861de46115748d5b716b4f4774f189a, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.577303 2015 consensus_queue.cc:260] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.577544 2015 ts_tablet_manager.cc:1428] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:24:58.579507 2013 tablet_bootstrap.cc:492] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: No bootstrap required, opened a new log
I20260620 12:24:58.579604 2013 ts_tablet_manager.cc:1397] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260620 12:24:58.580237 2013 raft_consensus.cc:348] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.580367 2013 raft_consensus.cc:374] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.580407 2013 raft_consensus.cc:729] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c98c7d9253ca484c8f9036a7af3c784c, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.580541 2013 consensus_queue.cc:260] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.580807 2013 ts_tablet_manager.cc:1428] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260620 12:24:58.628612 1991 tablet.cc:2369] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:24:58.630709 2019 raft_consensus.cc:482] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:24:58.630862 2019 raft_consensus.cc:504] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.631930 2019 leader_election.cc:288] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421), b45f21120b674406981e0b4e17c66d51 (127.0.98.3:46373)
I20260620 12:24:58.636674 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.632139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37421 (local address 127.0.98.2:49533)
0620 12:24:58.632337 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.632352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.632448 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.632741 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.632744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.632762 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:58.632880 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.632886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.633917 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.633920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.635266 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.635289 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.636147 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.636151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.636264 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.636296 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.636365 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.636538 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":29}
I20260620 12:24:58.636690 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.632233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49533 (local address 127.0.98.1:37421)
0620 12:24:58.632356 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.632362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.632399 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:24:58.632567 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.632573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.632630 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.632718 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:58.633009 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.633795 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.635425 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.636009 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.636151 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.636242 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.636431 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.636572 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:24:58.636708 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.632316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37515 (local address 127.0.98.3:46373)
0620 12:24:58.632425 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.632429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.632948 (+ 519us) server_negotiation.cc:407] Connection header received
0620 12:24:58.633064 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.633070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.633119 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.633203 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:24:58.633463 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.634020 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.635420 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.636009 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.636095 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.636170 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.636544 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.636613 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:24:58.637081 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.632152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46373 (local address 127.0.98.2:37515)
0620 12:24:58.632856 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.632874 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.632980 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.633211 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.633214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.633228 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:24:58.633361 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.633366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.634138 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.634141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.635297 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.635302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.636264 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.636268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.636406 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.636430 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.636827 (+ 397us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.636922 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":287,"thread_start_us":112,"threads_started":1}
I20260620 12:24:58.637316 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c1a37bb1254fdfafe46050dbb80a23" candidate_uuid: "d861de46115748d5b716b4f4774f189a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c98c7d9253ca484c8f9036a7af3c784c" is_pre_election: true
I20260620 12:24:58.637470 1664 raft_consensus.cc:2454] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d861de46115748d5b716b4f4774f189a in term 0.
I20260620 12:24:58.637537 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c1a37bb1254fdfafe46050dbb80a23" candidate_uuid: "d861de46115748d5b716b4f4774f189a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45f21120b674406981e0b4e17c66d51" is_pre_election: true
I20260620 12:24:58.637753 1944 raft_consensus.cc:2454] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d861de46115748d5b716b4f4774f189a in term 0.
I20260620 12:24:58.638075 1737 leader_election.cc:302] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c98c7d9253ca484c8f9036a7af3c784c, d861de46115748d5b716b4f4774f189a; no voters:
I20260620 12:24:58.638376 2019 raft_consensus.cc:2781] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:24:58.638456 2019 raft_consensus.cc:482] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:24:58.638509 2019 raft_consensus.cc:3037] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:58.639496 2019 raft_consensus.cc:504] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.639849 2019 leader_election.cc:288] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [CANDIDATE]: Term 1 election: Requested vote from peers c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421), b45f21120b674406981e0b4e17c66d51 (127.0.98.3:46373)
I20260620 12:24:58.640174 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c1a37bb1254fdfafe46050dbb80a23" candidate_uuid: "d861de46115748d5b716b4f4774f189a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45f21120b674406981e0b4e17c66d51"
I20260620 12:24:58.640199 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c1a37bb1254fdfafe46050dbb80a23" candidate_uuid: "d861de46115748d5b716b4f4774f189a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c98c7d9253ca484c8f9036a7af3c784c"
I20260620 12:24:58.640290 1944 raft_consensus.cc:3037] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:58.640317 1664 raft_consensus.cc:3037] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:58.641196 1664 raft_consensus.cc:2454] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d861de46115748d5b716b4f4774f189a in term 1.
I20260620 12:24:58.641386 1944 raft_consensus.cc:2454] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d861de46115748d5b716b4f4774f189a in term 1.
I20260620 12:24:58.641465 1737 leader_election.cc:302] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c98c7d9253ca484c8f9036a7af3c784c, d861de46115748d5b716b4f4774f189a; no voters:
I20260620 12:24:58.641631 2019 raft_consensus.cc:2781] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:58.641831 2019 raft_consensus.cc:686] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 1 LEADER]: Becoming Leader. State: Replica: d861de46115748d5b716b4f4774f189a, State: Running, Role: LEADER
I20260620 12:24:58.642068 2019 consensus_queue.cc:237] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.644354 1507 catalog_manager.cc:5495] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a reported cstate change: term changed from 0 to 1, leader changed from <none> to d861de46115748d5b716b4f4774f189a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d861de46115748d5b716b4f4774f189a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } health_report { overall_health: UNKNOWN } } }
W20260620 12:24:58.650986 1851 tablet.cc:2369] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:24:58.691043 1711 tablet.cc:2369] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:24:58 dist-test-slave-bwpp krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:58.697356 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.688635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.0.98.62:42461)
0620 12:24:58.688778 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:58.688783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:58.688974 (+ 191us) server_negotiation.cc:407] Connection header received
0620 12:24:58.689125 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:58.689128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:58.689169 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:58.689251 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:58.689884 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.690414 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.691323 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.691534 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.691637 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.694273 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:58.694277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:58.694279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:58.694298 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:58.695693 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.696147 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.696153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.696155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.696227 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:58.696491 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:58.696497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:58.696498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:58.696666 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:58.696760 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:58.697049 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 12:24:58.697207 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":43}
I20260620 12:24:58.697364 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:58.688578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.0.1:33194)
0620 12:24:58.688891 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:58.688903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:58.689033 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:58.689266 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:58.689269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:58.689678 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:58.689768 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.689774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.690516 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.690522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.691184 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:58.691197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:58.691649 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:58.691652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:58.691816 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:58.692388 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:58.692400 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:58.694137 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:58.695816 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.695826 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.695833 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.696055 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.696316 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:58.696319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:58.696321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:58.696410 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:58.696770 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:58.696778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:58.696889 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 12:24:58.696997 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:24:58.697157 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":34,"thread_start_us":121,"threads_started":1}
I20260620 12:24:58.701930 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33194:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:24:58.702817 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:24:58.707719 1644 tablet_service.cc:1467] Processing CreateTablet for tablet a75e804f203d40f8b48193f50e4f6c99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b695cdb3a33742b4a7d5d794adb16eca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:24:58.707777 1784 tablet_service.cc:1467] Processing CreateTablet for tablet a75e804f203d40f8b48193f50e4f6c99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b695cdb3a33742b4a7d5d794adb16eca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:24:58.708014 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a75e804f203d40f8b48193f50e4f6c99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b695cdb3a33742b4a7d5d794adb16eca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:24:58.708178 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75e804f203d40f8b48193f50e4f6c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.708187 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75e804f203d40f8b48193f50e4f6c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.708321 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75e804f203d40f8b48193f50e4f6c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:58.711205 2014 tablet_bootstrap.cc:492] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: Bootstrap starting.
I20260620 12:24:58.711284 2015 tablet_bootstrap.cc:492] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: Bootstrap starting.
I20260620 12:24:58.711903 2013 tablet_bootstrap.cc:492] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: Bootstrap starting.
I20260620 12:24:58.712102 2015 tablet_bootstrap.cc:654] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.712103 2014 tablet_bootstrap.cc:654] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.712881 2013 tablet_bootstrap.cc:654] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:58.713793 2014 tablet_bootstrap.cc:492] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: No bootstrap required, opened a new log
I20260620 12:24:58.713876 2014 ts_tablet_manager.cc:1397] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:24:58.714041 2013 tablet_bootstrap.cc:492] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: No bootstrap required, opened a new log
I20260620 12:24:58.714118 2013 ts_tablet_manager.cc:1397] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:58.714274 2014 raft_consensus.cc:348] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.714390 2014 raft_consensus.cc:374] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.714424 2014 raft_consensus.cc:729] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45f21120b674406981e0b4e17c66d51, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.714524 2013 raft_consensus.cc:348] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.714547 2014 consensus_queue.cc:260] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.714617 2013 raft_consensus.cc:374] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.714674 2013 raft_consensus.cc:729] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c98c7d9253ca484c8f9036a7af3c784c, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.714836 2014 ts_tablet_manager.cc:1428] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:24:58.714788 2013 consensus_queue.cc:260] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.715051 2013 ts_tablet_manager.cc:1428] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:24:58.715042 2023 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:58.715173 2023 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:24:58.715559 2020 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:58.715705 2020 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:24:58.716504 2015 tablet_bootstrap.cc:492] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: No bootstrap required, opened a new log
I20260620 12:24:58.716605 2015 ts_tablet_manager.cc:1397] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 12:24:58.717067 2015 raft_consensus.cc:348] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.717176 2015 raft_consensus.cc:374] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:58.717216 2015 raft_consensus.cc:729] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d861de46115748d5b716b4f4774f189a, State: Initialized, Role: FOLLOWER
I20260620 12:24:58.717335 2015 consensus_queue.cc:260] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:58.717592 2015 ts_tablet_manager.cc:1428] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:24:58.717765 2019 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:58.717875 2019 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:24:59.027179 2023 raft_consensus.cc:482] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:24:59.027324 2023 raft_consensus.cc:504] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:59.027771 2023 leader_election.cc:288] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421), d861de46115748d5b716b4f4774f189a (127.0.98.2:45721)
I20260620 12:24:59.028388 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75e804f203d40f8b48193f50e4f6c99" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d861de46115748d5b716b4f4774f189a" is_pre_election: true
I20260620 12:24:59.028560 1804 raft_consensus.cc:2454] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 0.
I20260620 12:24:59.028901 1877 leader_election.cc:302] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45f21120b674406981e0b4e17c66d51, d861de46115748d5b716b4f4774f189a; no voters:
I20260620 12:24:59.028900 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75e804f203d40f8b48193f50e4f6c99" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c98c7d9253ca484c8f9036a7af3c784c" is_pre_election: true
I20260620 12:24:59.029057 1664 raft_consensus.cc:2454] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 0.
I20260620 12:24:59.029106 2023 raft_consensus.cc:2781] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:24:59.029153 2023 raft_consensus.cc:482] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:24:59.029204 2023 raft_consensus.cc:3037] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:59.030368 2023 raft_consensus.cc:504] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:59.030743 2023 leader_election.cc:288] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 election: Requested vote from peers c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421), d861de46115748d5b716b4f4774f189a (127.0.98.2:45721)
I20260620 12:24:59.031134 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75e804f203d40f8b48193f50e4f6c99" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c98c7d9253ca484c8f9036a7af3c784c"
I20260620 12:24:59.031159 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75e804f203d40f8b48193f50e4f6c99" candidate_uuid: "b45f21120b674406981e0b4e17c66d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d861de46115748d5b716b4f4774f189a"
I20260620 12:24:59.031288 1664 raft_consensus.cc:3037] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:59.031299 1804 raft_consensus.cc:3037] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:59.032232 1664 raft_consensus.cc:2454] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 1.
I20260620 12:24:59.032531 1877 leader_election.cc:302] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b45f21120b674406981e0b4e17c66d51, c98c7d9253ca484c8f9036a7af3c784c; no voters:
I20260620 12:24:59.032763 1804 raft_consensus.cc:2454] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45f21120b674406981e0b4e17c66d51 in term 1.
I20260620 12:24:59.032828 2023 raft_consensus.cc:2781] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:59.032927 2023 raft_consensus.cc:686] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [term 1 LEADER]: Becoming Leader. State: Replica: b45f21120b674406981e0b4e17c66d51, State: Running, Role: LEADER
I20260620 12:24:59.033073 2023 consensus_queue.cc:237] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } }
I20260620 12:24:59.033754 2021 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: TxnStatusTablet state changed. Reason: New leader b45f21120b674406981e0b4e17c66d51. Latest consensus state: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:59.033855 2021 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:24:59.034139 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:24:59.034382 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:24:59.034958 1507 catalog_manager.cc:5495] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b45f21120b674406981e0b4e17c66d51 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } health_report { overall_health: HEALTHY } } }
I20260620 12:24:59.086591 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.082200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46373 (local address 127.0.0.1:51380)
0620 12:24:59.082348 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:59.082364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:59.082502 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:59.082858 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:59.082861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:59.082869 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:24:59.082985 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:59.082991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.084084 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.084088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:59.085078 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:59.085088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.085754 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.085758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:59.085851 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.085918 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:24:59.086261 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:24:59.086282 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:24:59.086342 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 12:24:59.086437 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 12:24:59.086627 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.082259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.98.3:46373)
0620 12:24:59.082382 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:59.082386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:59.082408 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:24:59.082634 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:59.082637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:59.082741 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:59.082825 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:59.083096 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.083962 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.085201 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.085633 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.085757 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.086072 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:59.086167 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:59.086245 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:59.086364 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 12:24:59.086433 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:24:59.091149 1664 raft_consensus.cc:1261] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Refusing update from remote peer b45f21120b674406981e0b4e17c66d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:59.091148 1804 raft_consensus.cc:1261] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Refusing update from remote peer b45f21120b674406981e0b4e17c66d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:59.091555 2021 consensus_queue.cc:1035] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:59.091701 2023 consensus_queue.cc:1035] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:59.093185 2020 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: TxnStatusTablet state changed. Reason: New leader b45f21120b674406981e0b4e17c66d51. Latest consensus state: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:59.093307 2020 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:24:59.093411 2019 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: TxnStatusTablet state changed. Reason: New leader b45f21120b674406981e0b4e17c66d51. Latest consensus state: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:59.093511 2019 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:24:59.095577 2023 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:59.095685 2023 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:24:59.096053 2019 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:59.096247 2019 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P d861de46115748d5b716b4f4774f189a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:24:59.097810 2021 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:59.097921 2021 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P b45f21120b674406981e0b4e17c66d51: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:24:59.098517 2020 tablet_replica.cc:439] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b45f21120b674406981e0b4e17c66d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 } } peers { permanent_uuid: "d861de46115748d5b716b4f4774f189a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45721 } } peers { permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 } } }
I20260620 12:24:59.098614 2020 tablet_replica.cc:442] T a75e804f203d40f8b48193f50e4f6c99 P c98c7d9253ca484c8f9036a7af3c784c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:24:59.098448 2059 mvcc.cc:205] Tried to move back new op lower bound from 7298901193073098752 to 7298901192840851456. Current Snapshot: MvccSnapshot[applied={T|T < 7298901193073098752 or (T in {7298901193073098752})}]
I20260620 12:24:59.108398 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.104197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.0.98.2:45721)
0620 12:24:59.104396 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:59.104406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:59.104428 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:24:59.104578 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:59.104580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:59.104658 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:59.104790 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:59.105019 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.105547 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.106826 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.107074 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.107164 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.107438 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:59.107982 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:59.108080 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:59.108163 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 12:24:59.108210 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260620 12:24:59.119676 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.115808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46373 (local address 127.0.98.2:48175)
0620 12:24:59.116169 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:59.116183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:59.116284 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:59.116550 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:59.116553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:59.116561 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:24:59.116665 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:59.116671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.117373 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.117377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:59.118393 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:59.118404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.118869 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.118872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:59.118970 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.119022 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:24:59.119401 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:24:59.119407 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:24:59.119470 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:24:59.119536 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":101,"threads_started":1}
I20260620 12:24:59.119688 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.115896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48175 (local address 127.0.98.3:46373)
0620 12:24:59.116025 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:59.116030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:59.116250 (+ 220us) server_negotiation.cc:407] Connection header received
0620 12:24:59.116408 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:59.116411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:59.116453 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:59.116517 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:59.116770 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.117274 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.118504 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.118767 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.118850 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.119141 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:59.119238 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:59.119354 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:59.119507 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 12:24:59.119560 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:24:59.126435 1664 raft_consensus.cc:1261] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Refusing update from remote peer d861de46115748d5b716b4f4774f189a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:59.126546 1944 raft_consensus.cc:1261] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Refusing update from remote peer d861de46115748d5b716b4f4774f189a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:59.126927 2019 consensus_queue.cc:1035] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c98c7d9253ca484c8f9036a7af3c784c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:59.127100 2042 consensus_queue.cc:1035] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45f21120b674406981e0b4e17c66d51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:59.149549 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.145723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46443 (local address 127.0.98.2:45721)
0620 12:24:59.145842 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:59.145847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:59.146002 (+ 155us) server_negotiation.cc:407] Connection header received
0620 12:24:59.146136 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:59.146139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:59.146189 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:59.146253 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:59.146493 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.147257 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.148413 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.148713 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.148813 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.149130 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:59.149207 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:59.149280 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:59.149388 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 12:24:59.149437 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:24:59.149607 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.145610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45721 (local address 127.0.98.3:46443)
0620 12:24:59.145919 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:59.145935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:59.146034 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:59.146278 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:59.146281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:59.146288 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:24:59.146390 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:59.146396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.147380 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.147384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:59.148292 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:59.148303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.148834 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.148838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:59.148937 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.148994 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:24:59.149302 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:24:59.149309 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:24:59.149382 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 12:24:59.149462 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260620 12:24:59.191226 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:24:59.191413 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:24:59.194146 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df24dd3e196146759f850d6b4a6e1183: Sending DeleteTablet for 3 replicas of tablet 31c1a37bb1254fdfafe46050dbb80a23
I20260620 12:24:59.194757 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c1a37bb1254fdfafe46050dbb80a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:24:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260620 12:24:59.194780 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c1a37bb1254fdfafe46050dbb80a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:24:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38720
I20260620 12:24:59.195003 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c1a37bb1254fdfafe46050dbb80a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:24:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51370
I20260620 12:24:59.195215 2079 tablet_replica.cc:330] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: stopping tablet replica
I20260620 12:24:59.195398 2081 tablet_replica.cc:330] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: stopping tablet replica
I20260620 12:24:59.195396 2079 raft_consensus.cc:2229] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:24:59.195546 2081 raft_consensus.cc:2229] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:24:59.195562 2079 raft_consensus.cc:2258] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:24:59.195694 2081 raft_consensus.cc:2258] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:24:59.196327 2079 ts_tablet_manager.cc:1905] T 31c1a37bb1254fdfafe46050dbb80a23 P c98c7d9253ca484c8f9036a7af3c784c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:24:59.196518 2081 ts_tablet_manager.cc:1905] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:24:59.196739 2080 tablet_replica.cc:330] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: stopping tablet replica
I20260620 12:24:59.196864 2080 raft_consensus.cc:2229] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:24:59.197189 2080 raft_consensus.cc:2258] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:24:59.198135 2080 ts_tablet_manager.cc:1905] T 31c1a37bb1254fdfafe46050dbb80a23 P d861de46115748d5b716b4f4774f189a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 12:24:59 dist-test-slave-bwpp krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:24:59.209479 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.196505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.0.98.1:37421)
0620 12:24:59.198377 (+ 1872us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:59.198382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:59.198612 (+ 230us) server_negotiation.cc:407] Connection header received
0620 12:24:59.198756 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:59.198759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:59.198809 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:59.198909 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:59.199913 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.200754 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.201696 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.202056 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.202161 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.205170 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:59.205183 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:59.205201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:59.205229 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:59.207686 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:59.208136 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:59.208143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:59.208145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:59.208238 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:59.208491 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:59.208495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:59.208497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:59.208797 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:59.209017 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:59.209137 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 12:24:59.209266 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1796,"thread_start_us":76,"threads_started":1}
W20260620 12:24:59.209919 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58532
I20260620 12:24:59.220794 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.211761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.0.98.62:42461)
0620 12:24:59.212021 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:59.212024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:59.212038 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:24:59.212129 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:59.212131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:59.212171 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:59.212277 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:59.212958 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.213831 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.214595 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:59.214977 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:59.215076 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:59.216703 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:59.216707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:59.216710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:59.216737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:59.218343 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:59.218779 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:59.218782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:59.218784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:59.218893 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:59.219163 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:59.219166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:59.219168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:59.219836 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:59.219981 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:59.220488 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 12:24:59.220614 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":114,"threads_started":1}
W20260620 12:24:59.221230 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33202
I20260620 12:24:59.222559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 1579
I20260620 12:24:59.251112 2081 ts_tablet_manager.cc:1918] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:24:59.251224 2081 log.cc:1196] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/31c1a37bb1254fdfafe46050dbb80a23
I20260620 12:24:59.251542 2081 ts_tablet_manager.cc:1939] T 31c1a37bb1254fdfafe46050dbb80a23 P b45f21120b674406981e0b4e17c66d51: Deleting consensus metadata
W20260620 12:24:59.252969 1493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37421 (error 108)
I20260620 12:24:59.254009 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.253680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37421 (local address 127.0.0.1:58540)
0620 12:24:59.253866 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:59.253922 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":53,"threads_started":1}
I20260620 12:24:59.254175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 1719
W20260620 12:24:59.255003 1493 catalog_manager.cc:4584] TS c98c7d9253ca484c8f9036a7af3c784c (127.0.98.1:37421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31c1a37bb1254fdfafe46050dbb80a23: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37421: connect: Connection refused (error 111)
I20260620 12:24:59.279409 1494 catalog_manager.cc:4841] TS b45f21120b674406981e0b4e17c66d51 (127.0.98.3:46373): tablet 31c1a37bb1254fdfafe46050dbb80a23 (table test-table [id=0e566b21513e4737a29bc751289a76e1]) successfully deleted
I20260620 12:24:59.284365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 1859
I20260620 12:24:59.286258 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:59.285378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45721 (local address 127.0.0.1:38740)
0620 12:24:59.286070 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:59.286150 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570}
I20260620 12:24:59.290944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 1475
2026-06-20T12:24:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2994 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:59 dist-test-slave-bwpp krb5kdc[2092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:59 dist-test-slave-bwpp krb5kdc[2092](info): set up 2 sockets
Jun 20 12:24:59 dist-test-slave-bwpp krb5kdc[2092](info): commencing operation
krb5kdc: starting...
W20260620 12:25:00.434510 392 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:00 dist-test-slave-bwpp krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:00.551549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40507
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:00.659250 2105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:00.659551 2105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:00.659622 2105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:00.663592 2105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:00.663677 2105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:25:00.663712 2105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:00.663749 2105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:00.663777 2105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:00.663815 2105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:00.668965 2105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40507
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:00.670018 2105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:00.671051 2105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:00.679255 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:00.679283 2111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:00.679303 2110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:00.679354 2105 server_base.cc:1034] running on GCE node
I20260620 12:25:00.680048 2105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:00.681062 2105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:00.682216 2105 hybrid_clock.cc:648] HybridClock initialized: now 1781958300682192 us; error 37 us; skew 500 ppm
Jun 20 12:25:00 dist-test-slave-bwpp krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:00.684962 2105 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:00.685778 2105 webserver.cc:460] Webserver started at http://127.0.98.62:46021/ using document root <none> and password file <none>
I20260620 12:25:00.686295 2105 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:00.686383 2105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:00.686600 2105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:00.688647 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "cc8336a847774b2d89f71730029d88f0"
format_stamp: "Formatted at 2026-06-20 12:25:00 on dist-test-slave-bwpp"
server_key: "9b519ea84e9aa721a0b8ecfb75758ef3"
server_key_iv: "ca091b058bd6be8926ec489823fa360e"
server_key_version: "encryptionkey@0"
I20260620 12:25:00.689205 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc8336a847774b2d89f71730029d88f0"
format_stamp: "Formatted at 2026-06-20 12:25:00 on dist-test-slave-bwpp"
server_key: "9b519ea84e9aa721a0b8ecfb75758ef3"
server_key_iv: "ca091b058bd6be8926ec489823fa360e"
server_key_version: "encryptionkey@0"
I20260620 12:25:00.692830 2105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:00.695200 2120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:00.696410 2105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:00.696556 2105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "cc8336a847774b2d89f71730029d88f0"
format_stamp: "Formatted at 2026-06-20 12:25:00 on dist-test-slave-bwpp"
server_key: "9b519ea84e9aa721a0b8ecfb75758ef3"
server_key_iv: "ca091b058bd6be8926ec489823fa360e"
server_key_version: "encryptionkey@0"
I20260620 12:25:00.696677 2105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:00.723616 2105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:00.727005 2105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:00.727206 2105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:00.735226 2105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40507
I20260620 12:25:00.735241 2182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40507 every 8 connection(s)
I20260620 12:25:00.736120 2105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:00.737222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 2105
I20260620 12:25:00.737365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:00.737636 392 external_mini_cluster.cc:1428] Setting key b17bb48264b08d0b8a92c6d15f5fa4d9
I20260620 12:25:00.739173 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:00.744333 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Bootstrap starting.
Jun 20 12:25:00 dist-test-slave-bwpp krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:00.747025 2183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:00.748027 2183 log.cc:824] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:00.749778 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: No bootstrap required, opened a new log
I20260620 12:25:00.751374 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:00.739032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.98.62:40507)
0620 12:25:00.739424 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:00.739440 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:00.739475 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:25:00.740105 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:00.740120 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:00.740432 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:00.740773 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:00.741581 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:00.742551 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:00.743421 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:00.743885 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:00.744057 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:00.746825 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:00.746831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:00.746843 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:00.746870 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:00.748602 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:00.749046 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:00.749051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:00.749055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:00.749166 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:00.749445 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:00.749448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:00.749450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:00.749831 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:00.750030 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:00.750315 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 12:25:00.750623 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":132,"threads_started":1}
I20260620 12:25:00.753124 2183 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8336a847774b2d89f71730029d88f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40507 } }
I20260620 12:25:00.753407 2183 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:00.753526 2183 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8336a847774b2d89f71730029d88f0, State: Initialized, Role: FOLLOWER
I20260620 12:25:00.753943 2183 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8336a847774b2d89f71730029d88f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40507 } }
I20260620 12:25:00.754081 2183 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:00.754145 2183 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:00.754241 2183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:00.755178 2183 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8336a847774b2d89f71730029d88f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40507 } }
I20260620 12:25:00.755505 2183 leader_election.cc:302] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8336a847774b2d89f71730029d88f0; no voters:
I20260620 12:25:00.755800 2183 leader_election.cc:288] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:00.755908 2188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:00.756134 2188 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [term 1 LEADER]: Becoming Leader. State: Replica: cc8336a847774b2d89f71730029d88f0, State: Running, Role: LEADER
I20260620 12:25:00.756456 2188 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8336a847774b2d89f71730029d88f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40507 } }
I20260620 12:25:00.756978 2183 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:00.758119 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8336a847774b2d89f71730029d88f0. Latest consensus state: current_term: 1 leader_uuid: "cc8336a847774b2d89f71730029d88f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8336a847774b2d89f71730029d88f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40507 } } }
I20260620 12:25:00.758255 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:00.758598 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:00.758986 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc8336a847774b2d89f71730029d88f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8336a847774b2d89f71730029d88f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40507 } } }
I20260620 12:25:00.759133 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:00.761344 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:00.766510 2197 catalog_manager.cc:1347] Generated new cluster ID: 6c4c52f6a2be4d349503aae4b90011ae
I20260620 12:25:00.766583 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:00.774210 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:00.775411 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:00.780259 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Generated new TSK 0
I20260620 12:25:00.780875 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:25:00.809546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40507
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:00.922375 2209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:00.922629 2209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:00.922739 2209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:00.926832 2209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:00.926887 2209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:00.926924 2209 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:25:00.926939 2209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:25:00.926970 2209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:00.932018 2209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:00.933254 2209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:00.934312 2209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:00.943656 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:00.943646 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:00.943662 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:00.943871 2209 server_base.cc:1034] running on GCE node
I20260620 12:25:00.944226 2209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:00.944875 2209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:00.946043 2209 hybrid_clock.cc:648] HybridClock initialized: now 1781958300946028 us; error 26 us; skew 500 ppm
Jun 20 12:25:00 dist-test-slave-bwpp krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:00.948764 2209 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:00.949611 2209 webserver.cc:460] Webserver started at http://127.0.98.1:33627/ using document root <none> and password file <none>
I20260620 12:25:00.950134 2209 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:00.950196 2209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:00.950356 2209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:00.952327 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
format_stamp: "Formatted at 2026-06-20 12:25:00 on dist-test-slave-bwpp"
server_key: "92e6d1b7da21c3d3f0889dd1572acba1"
server_key_iv: "eec5762398ceeaa144aa7a5c63036662"
server_key_version: "encryptionkey@0"
I20260620 12:25:00.952831 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
format_stamp: "Formatted at 2026-06-20 12:25:00 on dist-test-slave-bwpp"
server_key: "92e6d1b7da21c3d3f0889dd1572acba1"
server_key_iv: "eec5762398ceeaa144aa7a5c63036662"
server_key_version: "encryptionkey@0"
I20260620 12:25:00.956565 2209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:00.959162 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:00.960364 2209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:00.960457 2209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
format_stamp: "Formatted at 2026-06-20 12:25:00 on dist-test-slave-bwpp"
server_key: "92e6d1b7da21c3d3f0889dd1572acba1"
server_key_iv: "eec5762398ceeaa144aa7a5c63036662"
server_key_version: "encryptionkey@0"
I20260620 12:25:00.960552 2209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:00.976313 2209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:00.979563 2209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:00.979717 2209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:00.981086 2209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:00.981134 2209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:00.981174 2209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:00.981189 2209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:25:00 dist-test-slave-bwpp krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:00.992092 2209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36383
I20260620 12:25:00.992134 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36383 every 8 connection(s)
I20260620 12:25:00.993188 2209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:00.994643 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:00.981799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41309 (local address 127.0.98.62:40507)
0620 12:25:00.981967 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:00.981973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:00.982806 (+ 833us) server_negotiation.cc:407] Connection header received
0620 12:25:00.983783 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:00.983787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:00.983853 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:00.983918 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:00.985568 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:00.986401 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:00.987615 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:00.987893 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:00.987996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:00.990719 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:00.990722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:00.990726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:00.990756 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:00.992460 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:00.992982 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:00.992993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:00.992995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:00.993088 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:00.993483 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:00.993488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:00.993490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:00.993734 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:00.993841 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:00.994411 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 12:25:00.994522 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:25:00.995471 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:00.982200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40507 (local address 127.0.98.1:41309)
0620 12:25:00.982630 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:00.982703 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:00.983534 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:00.984350 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:00.984361 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:00.984871 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:00.985418 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:00.985451 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:00.986521 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:00.986524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:00.987492 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:00.987502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:00.988035 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:00.988038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:00.988209 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:00.988825 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:00.988853 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:00.990536 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:00.992586 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:00.992592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:00.992604 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:00.992875 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:00.993188 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:00.993190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:00.993191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:00.993394 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:00.993897 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:00.993904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:00.994147 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 12:25:00.994303 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 12:25:00.994540 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":116,"threads_started":1}
I20260620 12:25:00.995591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 2209
I20260620 12:25:00.995698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:00.995975 392 external_mini_cluster.cc:1428] Setting key b8ccfb9df00be9f9daa2b7fb7d00e18b
I20260620 12:25:00.998713 2340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40507
I20260620 12:25:00.998860 2340 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:00.999322 2340 heartbeater.cc:511] Master 127.0.98.62:40507 requested a full tablet report, sending...
I20260620 12:25:01.000880 2136 ts_manager.cc:194] Registered new tserver with Master: 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383)
I20260620 12:25:01.002169 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41309
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:25:01.007783 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:00.999702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36129 (local address 127.0.98.62:40507)
0620 12:25:00.999830 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:00.999836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.000096 (+ 260us) server_negotiation.cc:407] Connection header received
0620 12:25:01.000219 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.000222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.000264 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.000381 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:01.000821 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.001572 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.002955 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.003257 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.003367 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.004734 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:01.004737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:01.004739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:01.004759 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:01.006176 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.006703 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.006707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.006709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.006791 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.007013 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.007015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.007016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.007159 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:01.007229 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.007582 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.007680 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:25:01.007786 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:00.999627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40507 (local address 127.0.98.1:36129)
0620 12:25:01.000017 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.000035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.000137 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.000400 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.000406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.000604 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:01.000696 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.000701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.001723 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.001730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.002810 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.002824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.003387 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.003393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.003525 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.004103 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:01.004119 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:01.004582 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:01.006292 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.006297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.006302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.006562 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.006873 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.006876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.006877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.006952 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.007261 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:01.007263 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:01.007425 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.007520 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.007633 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:01.023514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40507
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:01.130854 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:01.131145 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:01.131227 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:01.135308 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:01.135386 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:01.135455 2349 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:25:01.135495 2349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:25:01.135560 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:01.140740 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:01.141741 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:01.142771 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:01.151968 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:01.152127 2349 server_base.cc:1034] running on GCE node
W20260620 12:25:01.152032 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:01.151968 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:01.152697 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:01.153347 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:01.154529 2349 hybrid_clock.cc:648] HybridClock initialized: now 1781958301154498 us; error 43 us; skew 500 ppm
Jun 20 12:25:01 dist-test-slave-bwpp krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:01.160562 2349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:01.161494 2349 webserver.cc:460] Webserver started at http://127.0.98.2:34665/ using document root <none> and password file <none>
I20260620 12:25:01.162014 2349 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:01.162070 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:01.162225 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:01.164356 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "ceb87cd71b9140c18a608f029ad6b266"
format_stamp: "Formatted at 2026-06-20 12:25:01 on dist-test-slave-bwpp"
server_key: "969869d57d628bb9dd39c32e5d49c97e"
server_key_iv: "8f5f2007f634d07887c07335a6e1f5de"
server_key_version: "encryptionkey@0"
I20260620 12:25:01.164870 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ceb87cd71b9140c18a608f029ad6b266"
format_stamp: "Formatted at 2026-06-20 12:25:01 on dist-test-slave-bwpp"
server_key: "969869d57d628bb9dd39c32e5d49c97e"
server_key_iv: "8f5f2007f634d07887c07335a6e1f5de"
server_key_version: "encryptionkey@0"
I20260620 12:25:01.168511 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:01.170917 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:01.172116 2349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:01.172237 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "ceb87cd71b9140c18a608f029ad6b266"
format_stamp: "Formatted at 2026-06-20 12:25:01 on dist-test-slave-bwpp"
server_key: "969869d57d628bb9dd39c32e5d49c97e"
server_key_iv: "8f5f2007f634d07887c07335a6e1f5de"
server_key_version: "encryptionkey@0"
I20260620 12:25:01.172379 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:01.187599 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:01.191166 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:01.191390 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:01.192865 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:01.192938 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:01.193012 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:01.193063 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:25:01 dist-test-slave-bwpp krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:01.204562 2349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41281
I20260620 12:25:01.204632 2479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41281 every 8 connection(s)
I20260620 12:25:01.205693 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:01.207094 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.193687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33311 (local address 127.0.98.62:40507)
0620 12:25:01.193861 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.193868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.194692 (+ 824us) server_negotiation.cc:407] Connection header received
0620 12:25:01.195684 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.195687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.195742 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.195820 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:01.197419 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.198315 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.199681 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.200018 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.200121 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.203250 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:01.203255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:01.203257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:01.203286 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:01.204882 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.205457 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.205459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.205461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.205534 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.205900 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.205903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.205904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.206129 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:01.206227 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.206854 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.206970 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:25:01.207852 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.194070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40507 (local address 127.0.98.2:33311)
0620 12:25:01.194515 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.194566 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.195478 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.196134 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.196143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.196639 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:01.197280 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.197299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.198480 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.198483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.199558 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.199568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.200169 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.200173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.200402 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.200979 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:01.201002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:01.203054 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:01.204989 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.204996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.205008 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.205353 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.205620 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.205623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.205625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.205810 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.206277 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:01.206283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:01.206564 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.206763 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.207017 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":121,"threads_started":1}
I20260620 12:25:01.209158 2480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40507
I20260620 12:25:01.209360 2480 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:01.209410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 2349
I20260620 12:25:01.209523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:01.209828 392 external_mini_cluster.cc:1428] Setting key bcb243ff5748a193f713e9047763e354
I20260620 12:25:01.209919 2480 heartbeater.cc:511] Master 127.0.98.62:40507 requested a full tablet report, sending...
I20260620 12:25:01.210973 2136 ts_manager.cc:194] Registered new tserver with Master: ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:01.211735 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33311
I20260620 12:25:01.219199 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.211354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40507 (local address 127.0.98.2:36919)
0620 12:25:01.211634 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.211649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.211731 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.212002 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.212008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.212240 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:01.212392 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.212403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.213420 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.213424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.214132 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.214142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.214819 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.214822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.214895 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.215411 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:01.215422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:01.215734 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:01.217675 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.217678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.217680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.217879 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.218156 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.218158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.218159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.218230 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.218767 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:01.218769 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:01.218852 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.218952 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.219063 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260620 12:25:01.219345 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.211663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36919 (local address 127.0.98.62:40507)
0620 12:25:01.211765 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.211772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.211788 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:25:01.211827 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.211830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.211868 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.211951 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:01.212520 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.213253 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.214264 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.214713 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.214831 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.215866 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:01.215870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:01.215873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:01.215899 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:01.217554 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.217972 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.217976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.217978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.218066 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.218311 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.218316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.218317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.218623 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:01.218764 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.219035 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.219149 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:01.237640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40507
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:01.345955 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:01.346197 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:01.346243 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:01.350216 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:01.350270 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:01.350308 2489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:25:01.350322 2489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:25:01.350353 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:01.355474 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:01.356366 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:01.357311 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:01.366530 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:01.366539 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:01.366592 2489 server_base.cc:1034] running on GCE node
W20260620 12:25:01.366534 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:01.367101 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:01.367681 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:01.368855 2489 hybrid_clock.cc:648] HybridClock initialized: now 1781958301368840 us; error 26 us; skew 500 ppm
Jun 20 12:25:01 dist-test-slave-bwpp krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:01.371481 2489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:01.372304 2489 webserver.cc:460] Webserver started at http://127.0.98.3:38385/ using document root <none> and password file <none>
I20260620 12:25:01.372886 2489 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:01.372970 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:01.373183 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:01.375242 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "3428959204db42e483dc5678276e4959"
format_stamp: "Formatted at 2026-06-20 12:25:01 on dist-test-slave-bwpp"
server_key: "e08944e2a2dc531e2a01f915267f83a2"
server_key_iv: "30dcdc6b758000148144b52873c3cf95"
server_key_version: "encryptionkey@0"
I20260620 12:25:01.375777 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3428959204db42e483dc5678276e4959"
format_stamp: "Formatted at 2026-06-20 12:25:01 on dist-test-slave-bwpp"
server_key: "e08944e2a2dc531e2a01f915267f83a2"
server_key_iv: "30dcdc6b758000148144b52873c3cf95"
server_key_version: "encryptionkey@0"
I20260620 12:25:01.379371 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:25:01.381641 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:01.383021 2489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 12:25:01.383137 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "3428959204db42e483dc5678276e4959"
format_stamp: "Formatted at 2026-06-20 12:25:01 on dist-test-slave-bwpp"
server_key: "e08944e2a2dc531e2a01f915267f83a2"
server_key_iv: "30dcdc6b758000148144b52873c3cf95"
server_key_version: "encryptionkey@0"
I20260620 12:25:01.383236 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:01.398428 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:01.401382 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:01.401574 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:01.402951 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:01.403021 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:01.403082 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:01.403118 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:25:01 dist-test-slave-bwpp krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:01.413932 2489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35423
I20260620 12:25:01.414336 2619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35423 every 8 connection(s)
I20260620 12:25:01.415066 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:01.416824 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.403686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33227 (local address 127.0.98.62:40507)
0620 12:25:01.403801 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.403804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.404964 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 12:25:01.405962 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.405968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.406034 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.406094 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:01.407546 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.408451 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.409844 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.410280 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.410413 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.413114 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:01.413118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:01.413121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:01.413171 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:01.414895 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.415421 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.415426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.415431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.415523 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.415915 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.415918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.415919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.416089 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:01.416169 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.416597 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.416694 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 12:25:01.417572 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.404318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40507 (local address 127.0.98.3:33227)
0620 12:25:01.404817 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.404858 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.405702 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.406442 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.406448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.406818 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:01.407376 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.407396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.408594 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.408600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.409742 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.409751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.410466 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.410473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.410712 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.411332 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:01.411350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:01.412940 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:01.414994 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.414998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.415009 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.415304 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.415639 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.415644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.415649 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.415830 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.416178 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:01.416182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:01.416387 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.416532 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.416753 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
I20260620 12:25:01.418781 2620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40507
I20260620 12:25:01.418938 2620 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:01.419458 2620 heartbeater.cc:511] Master 127.0.98.62:40507 requested a full tablet report, sending...
I20260620 12:25:01.420629 2135 ts_manager.cc:194] Registered new tserver with Master: 3428959204db42e483dc5678276e4959 (127.0.98.3:35423)
I20260620 12:25:01.421381 2135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33227
I20260620 12:25:01.423537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 2489
I20260620 12:25:01.423645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:01.423926 392 external_mini_cluster.cc:1428] Setting key caa36ec888f67934002bd33f0c55a988
I20260620 12:25:01.425705 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:25:01.428380 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.420998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40507 (local address 127.0.98.3:37215)
0620 12:25:01.421296 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.421309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.421427 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.421702 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.421704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.421916 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:01.422015 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.422021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.423094 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.423096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.423899 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.423909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.424422 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.424426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.424518 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.425017 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:01.425034 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:01.425399 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:01.426963 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.426968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.426970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.427157 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.427485 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.427489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.427491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.427588 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.428004 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:01.428006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:01.428074 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.428151 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.428246 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":75,"threads_started":1}
I20260620 12:25:01.428542 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.421059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37215 (local address 127.0.98.62:40507)
0620 12:25:01.421170 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.421173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.421386 (+ 213us) server_negotiation.cc:407] Connection header received
0620 12:25:01.421519 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.421522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.421563 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.421656 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:01.422120 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.422882 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.424037 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.424250 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.424420 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.425526 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:01.425530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:01.425532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:01.425555 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:01.426862 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.427251 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.427256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.427257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.427367 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.427677 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.427683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.427685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.427906 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:01.428012 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.428232 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.428360 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 12:25:01 dist-test-slave-bwpp krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:25:01 dist-test-slave-bwpp krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:01.450448 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.443088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.0.98.62:40507)
0620 12:25:01.443206 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.443210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.443222 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:25:01.443258 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.443261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.443295 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.443367 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:01.444002 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.444524 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.445337 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.445538 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.445613 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.447954 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:01.447957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:01.447959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:01.447981 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:01.449171 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.449523 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.449525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.449527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.449594 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.449805 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.449807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.449808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.449956 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:01.450063 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.450231 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.450320 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:25:01.453219 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:01.456058 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:01.470037 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.464645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.0.98.3:35423)
0620 12:25:01.465843 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.465856 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.465878 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:25:01.465951 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.465959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.466135 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.466283 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:01.466489 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.467352 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.468860 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.469363 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.469482 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.469655 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.469791 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.469877 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":117,"threads_started":1}
I20260620 12:25:01.470029 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.464183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35423 (local address 127.0.0.1:38306)
0620 12:25:01.465188 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.465200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.465270 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.466255 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.466258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.466273 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:01.466374 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.466380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.467460 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.467463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.468718 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.468728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.469482 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.469486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.469611 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.469653 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.469759 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.469873 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":86,"thread_start_us":82,"threads_started":1}
I20260620 12:25:01.470248 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.463880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36383 (local address 127.0.0.1:33280)
0620 12:25:01.464800 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.464866 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.464973 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.465548 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.465555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.465584 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:01.465915 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.465920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.467246 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.467255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.468718 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.468728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.469836 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.469842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.469943 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.469961 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.470047 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.470114 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":149,"thread_start_us":72,"threads_started":1}
I20260620 12:25:01.470306 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.464012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.0.98.1:36383)
0620 12:25:01.464400 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.464414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.465106 (+ 692us) server_negotiation.cc:407] Connection header received
0620 12:25:01.465190 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.465196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.465402 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.465564 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:01.466027 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.467125 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.468847 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.469607 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.469701 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.469852 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.470077 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.470170 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
I20260620 12:25:01.470945 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.464688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.0.98.2:41281)
0620 12:25:01.465224 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.465235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.465253 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:25:01.465313 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.465317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.465459 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.465593 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:01.465845 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.467077 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.468597 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.469369 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.469590 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.469761 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.470569 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.470684 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":113,"threads_started":1}
I20260620 12:25:01.471342 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.463923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41281 (local address 127.0.0.1:54584)
0620 12:25:01.464798 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.464845 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.464959 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.465595 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.465598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.465620 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:01.465734 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.465741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.467186 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.467189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.468480 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.468488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.469509 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.469513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.470359 (+ 846us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.470383 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.470446 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.471204 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":761,"spinlock_wait_cycles":1408,"thread_start_us":101,"threads_started":1}
I20260620 12:25:01.473031 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 1c27978f6e4345cfb33fead3a773a591 (DEFAULT_TABLE table=test-table [id=e5ce45a998664fd89ec2bbf41caf5e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:01.473134 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 1c27978f6e4345cfb33fead3a773a591 (DEFAULT_TABLE table=test-table [id=e5ce45a998664fd89ec2bbf41caf5e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:01.473997 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c27978f6e4345cfb33fead3a773a591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.473695 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 1c27978f6e4345cfb33fead3a773a591 (DEFAULT_TABLE table=test-table [id=e5ce45a998664fd89ec2bbf41caf5e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:01.474117 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c27978f6e4345cfb33fead3a773a591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.474565 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c27978f6e4345cfb33fead3a773a591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.480736 2643 tablet_bootstrap.cc:492] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Bootstrap starting.
I20260620 12:25:01.482070 2644 tablet_bootstrap.cc:492] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Bootstrap starting.
I20260620 12:25:01.482923 2645 tablet_bootstrap.cc:492] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Bootstrap starting.
I20260620 12:25:01.483392 2643 tablet_bootstrap.cc:654] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.484066 2644 tablet_bootstrap.cc:654] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.484266 2643 log.cc:824] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:01.484829 2645 tablet_bootstrap.cc:654] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.484872 2644 log.cc:824] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:01.485603 2645 log.cc:824] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:01.486336 2643 tablet_bootstrap.cc:492] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: No bootstrap required, opened a new log
I20260620 12:25:01.486523 2643 ts_tablet_manager.cc:1397] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:25:01.486822 2644 tablet_bootstrap.cc:492] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: No bootstrap required, opened a new log
I20260620 12:25:01.486984 2644 ts_tablet_manager.cc:1397] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:01.487378 2645 tablet_bootstrap.cc:492] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: No bootstrap required, opened a new log
I20260620 12:25:01.487563 2645 ts_tablet_manager.cc:1397] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:01.490378 2643 raft_consensus.cc:348] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.490634 2643 raft_consensus.cc:374] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.490746 2643 raft_consensus.cc:729] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3428959204db42e483dc5678276e4959, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.490552 2645 raft_consensus.cc:348] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.490820 2645 raft_consensus.cc:374] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.490706 2644 raft_consensus.cc:348] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.490887 2645 raft_consensus.cc:729] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb87cd71b9140c18a608f029ad6b266, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.490943 2644 raft_consensus.cc:374] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.491001 2644 raft_consensus.cc:729] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a22cdf4c6b94e6ea11f91828e3f113e, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.491225 2643 consensus_queue.cc:260] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.491326 2645 consensus_queue.cc:260] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.491333 2644 consensus_queue.cc:260] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.492110 2340 heartbeater.cc:503] Master 127.0.98.62:40507 was elected leader, sending a full tablet report...
I20260620 12:25:01.492250 2645 ts_tablet_manager.cc:1428] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:01.492256 2643 ts_tablet_manager.cc:1428] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 12:25:01.492254 2644 ts_tablet_manager.cc:1428] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:25:01.492552 2480 heartbeater.cc:503] Master 127.0.98.62:40507 was elected leader, sending a full tablet report...
I20260620 12:25:01.492733 2620 heartbeater.cc:503] Master 127.0.98.62:40507 was elected leader, sending a full tablet report...
W20260620 12:25:01.494318 2341 tablet.cc:2369] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:01.543422 2649 raft_consensus.cc:482] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:01.543646 2649 raft_consensus.cc:504] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.544791 2649 leader_election.cc:288] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:01.549580 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.545045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33635 (local address 127.0.98.1:36383)
0620 12:25:01.545172 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.545179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.545241 (+ 62us) server_negotiation.cc:407] Connection header received
0620 12:25:01.545389 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.545391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.545436 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.545533 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:01.545840 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.546622 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.548362 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.548901 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.549032 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.549091 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.549372 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.549452 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:25:01.549582 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.544967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36383 (local address 127.0.98.3:33635)
0620 12:25:01.545163 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.545178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.545277 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.545553 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.545558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.545577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:01.545700 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.545704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.546751 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.546754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.548194 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.548205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.549067 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.549071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.549227 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.549256 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.549327 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.549415 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":29}
I20260620 12:25:01.549607 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.545164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42795 (local address 127.0.98.2:41281)
0620 12:25:01.545294 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.545301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.545498 (+ 197us) server_negotiation.cc:407] Connection header received
0620 12:25:01.545608 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.545613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.545671 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.545729 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:01.545978 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.546755 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.548596 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.549061 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.549160 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.549230 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.549419 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.549502 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:01.549595 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.545043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41281 (local address 127.0.98.3:42795)
0620 12:25:01.545415 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.545425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.545524 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.545765 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.545768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.545789 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:01.545885 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.545889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.546944 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.546947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.548489 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.548497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.549158 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.549161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.549276 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.549293 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.549394 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.549456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":21,"thread_start_us":131,"threads_started":1}
I20260620 12:25:01.550284 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27978f6e4345cfb33fead3a773a591" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266" is_pre_election: true
I20260620 12:25:01.550383 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27978f6e4345cfb33fead3a773a591" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" is_pre_election: true
I20260620 12:25:01.550596 2434 raft_consensus.cc:2454] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:01.550699 2294 raft_consensus.cc:2454] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:01.551098 2506 leader_election.cc:302] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, ceb87cd71b9140c18a608f029ad6b266; no voters:
I20260620 12:25:01.551366 2649 raft_consensus.cc:2781] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:01.551448 2649 raft_consensus.cc:482] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:01.551491 2649 raft_consensus.cc:3037] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:01.552482 2649 raft_consensus.cc:504] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.552790 2649 leader_election.cc:288] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Requested vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:01.553102 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27978f6e4345cfb33fead3a773a591" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266"
I20260620 12:25:01.553076 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27978f6e4345cfb33fead3a773a591" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
I20260620 12:25:01.553213 2434 raft_consensus.cc:3037] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:01.553213 2294 raft_consensus.cc:3037] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:01.554423 2434 raft_consensus.cc:2454] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:01.554423 2294 raft_consensus.cc:2454] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:01.554720 2506 leader_election.cc:302] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, ceb87cd71b9140c18a608f029ad6b266; no voters:
I20260620 12:25:01.554877 2649 raft_consensus.cc:2781] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:01.555073 2649 raft_consensus.cc:686] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Becoming Leader. State: Replica: 3428959204db42e483dc5678276e4959, State: Running, Role: LEADER
I20260620 12:25:01.555408 2649 consensus_queue.cc:237] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.558617 2136 catalog_manager.cc:5495] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3428959204db42e483dc5678276e4959 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3428959204db42e483dc5678276e4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } health_report { overall_health: HEALTHY } } }
Jun 20 12:25:01 dist-test-slave-bwpp krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:01.600416 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.592339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40507 (local address 127.0.0.1:59466)
0620 12:25:01.592602 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.592614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.592744 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.592982 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.592984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.593408 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:01.593493 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.593499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.594222 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.594226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.594822 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.594831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.595245 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.595248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.595322 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.595908 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:01.595920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:01.597310 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:01.598983 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.598993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.598998 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.599211 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.599464 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:01.599485 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:01.599497 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:01.599587 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:01.599928 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:01.599932 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:01.600036 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.600120 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.600235 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260620 12:25:01.600425 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.592371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.0.98.62:40507)
0620 12:25:01.592526 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.592532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.592688 (+ 156us) server_negotiation.cc:407] Connection header received
0620 12:25:01.592837 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.592841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.592883 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.592964 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:01.593608 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.594112 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.594928 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.595141 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.595244 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.597458 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:01.597461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:01.597463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:01.597485 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:01.598882 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.599298 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.599301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.599302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.599385 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:01.599664 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:01.599667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:01.599668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:01.599842 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:01.599918 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.600176 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.600287 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":54}
I20260620 12:25:01.604393 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:25:01.605147 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:01.610002 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9ec8df8ed341b5bdaede545710536f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27b0203beecf4281abe5a29ada2940a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:25:01.609952 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9ec8df8ed341b5bdaede545710536f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27b0203beecf4281abe5a29ada2940a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:25:01.609817 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9ec8df8ed341b5bdaede545710536f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27b0203beecf4281abe5a29ada2940a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:25:01.610628 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9ec8df8ed341b5bdaede545710536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.610628 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9ec8df8ed341b5bdaede545710536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.610628 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9ec8df8ed341b5bdaede545710536f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.612993 2643 tablet_bootstrap.cc:492] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: Bootstrap starting.
I20260620 12:25:01.613466 2645 tablet_bootstrap.cc:492] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: Bootstrap starting.
I20260620 12:25:01.614037 2643 tablet_bootstrap.cc:654] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.614236 2644 tablet_bootstrap.cc:492] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: Bootstrap starting.
I20260620 12:25:01.614471 2645 tablet_bootstrap.cc:654] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.615119 2644 tablet_bootstrap.cc:654] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.615383 2643 tablet_bootstrap.cc:492] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: No bootstrap required, opened a new log
I20260620 12:25:01.615458 2643 ts_tablet_manager.cc:1397] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 12:25:01.615809 2643 raft_consensus.cc:348] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.615928 2643 raft_consensus.cc:374] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.615974 2643 raft_consensus.cc:729] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3428959204db42e483dc5678276e4959, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.616106 2643 consensus_queue.cc:260] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.616264 2644 tablet_bootstrap.cc:492] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: No bootstrap required, opened a new log
I20260620 12:25:01.616346 2644 ts_tablet_manager.cc:1397] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:01.616423 2643 ts_tablet_manager.cc:1428] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:25:01.616811 2644 raft_consensus.cc:348] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.616925 2644 raft_consensus.cc:374] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.616971 2644 raft_consensus.cc:729] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a22cdf4c6b94e6ea11f91828e3f113e, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.617101 2644 consensus_queue.cc:260] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.617393 2644 ts_tablet_manager.cc:1428] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:01.617352 2650 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.617467 2645 tablet_bootstrap.cc:492] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: No bootstrap required, opened a new log
I20260620 12:25:01.617481 2650 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:01.617547 2645 ts_tablet_manager.cc:1397] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 12:25:01.617566 2649 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.617671 2649 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:01.618014 2645 raft_consensus.cc:348] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.618129 2645 raft_consensus.cc:374] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.618161 2645 raft_consensus.cc:729] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb87cd71b9140c18a608f029ad6b266, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.618295 2645 consensus_queue.cc:260] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.618583 2651 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.618743 2651 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:01.619457 2645 ts_tablet_manager.cc:1428] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260620 12:25:01.665975 2621 tablet.cc:2369] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:25:01.706861 2481 tablet.cc:2369] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:01.786073 2651 raft_consensus.cc:482] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:01.786219 2651 raft_consensus.cc:504] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.787633 2651 leader_election.cc:288] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), 3428959204db42e483dc5678276e4959 (127.0.98.3:35423)
I20260620 12:25:01.791720 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.787981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57587 (local address 127.0.98.3:35423)
0620 12:25:01.788107 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.788112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.788323 (+ 211us) server_negotiation.cc:407] Connection header received
0620 12:25:01.788449 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.788452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.788510 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.788593 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:01.788855 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.789355 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.790418 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.791057 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.791177 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.791255 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.791496 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.791591 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:25:01.791767 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.787907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35423 (local address 127.0.98.2:57587)
0620 12:25:01.788229 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.788240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.788356 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.788604 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.788607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.788623 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:01.788749 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.788754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.789462 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.789467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.790294 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.790302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.791196 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.791201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.791324 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.791383 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.791463 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.791542 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":12,"thread_start_us":74,"threads_started":1}
I20260620 12:25:01.791813 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.787957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59451 (local address 127.0.98.1:36383)
0620 12:25:01.788086 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.788091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.788235 (+ 144us) server_negotiation.cc:407] Connection header received
0620 12:25:01.788393 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.788396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.788442 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.788521 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:01.788793 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.789293 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.790508 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.791131 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.791232 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.791287 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.791618 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.791689 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:25:01.792282 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.787836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36383 (local address 127.0.98.2:59451)
0620 12:25:01.788150 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.788163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.788289 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.788531 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.788534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.788549 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:01.788692 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.788698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.789415 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.789418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.790292 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.790301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.791347 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.791349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.791477 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.791501 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.792071 (+ 570us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.792150 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":81,"thread_start_us":89,"threads_started":1}
I20260620 12:25:01.792542 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9ec8df8ed341b5bdaede545710536f" candidate_uuid: "ceb87cd71b9140c18a608f029ad6b266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3428959204db42e483dc5678276e4959" is_pre_election: true
I20260620 12:25:01.792800 2574 raft_consensus.cc:2454] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb87cd71b9140c18a608f029ad6b266 in term 0.
I20260620 12:25:01.793292 2367 leader_election.cc:302] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, ceb87cd71b9140c18a608f029ad6b266; no voters:
I20260620 12:25:01.793572 2651 raft_consensus.cc:2781] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:01.793638 2651 raft_consensus.cc:482] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:01.793684 2651 raft_consensus.cc:3037] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:01.794461 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9ec8df8ed341b5bdaede545710536f" candidate_uuid: "ceb87cd71b9140c18a608f029ad6b266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" is_pre_election: true
I20260620 12:25:01.794612 2294 raft_consensus.cc:2454] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb87cd71b9140c18a608f029ad6b266 in term 0.
I20260620 12:25:01.796006 2651 raft_consensus.cc:504] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.796378 2651 leader_election.cc:288] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [CANDIDATE]: Term 1 election: Requested vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), 3428959204db42e483dc5678276e4959 (127.0.98.3:35423)
I20260620 12:25:01.796753 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9ec8df8ed341b5bdaede545710536f" candidate_uuid: "ceb87cd71b9140c18a608f029ad6b266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
I20260620 12:25:01.796897 2294 raft_consensus.cc:3037] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:01.797590 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9ec8df8ed341b5bdaede545710536f" candidate_uuid: "ceb87cd71b9140c18a608f029ad6b266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3428959204db42e483dc5678276e4959"
I20260620 12:25:01.797715 2574 raft_consensus.cc:3037] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:01.798185 2294 raft_consensus.cc:2454] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb87cd71b9140c18a608f029ad6b266 in term 1.
I20260620 12:25:01.798513 2365 leader_election.cc:302] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a22cdf4c6b94e6ea11f91828e3f113e, ceb87cd71b9140c18a608f029ad6b266; no voters:
I20260620 12:25:01.798727 2651 raft_consensus.cc:2781] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:01.798928 2651 raft_consensus.cc:686] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [term 1 LEADER]: Becoming Leader. State: Replica: ceb87cd71b9140c18a608f029ad6b266, State: Running, Role: LEADER
I20260620 12:25:01.799129 2574 raft_consensus.cc:2454] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb87cd71b9140c18a608f029ad6b266 in term 1.
I20260620 12:25:01.799284 2651 consensus_queue.cc:237] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.801620 2666 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: TxnStatusTablet state changed. Reason: New leader ceb87cd71b9140c18a608f029ad6b266. Latest consensus state: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.801811 2666 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:25:01.802155 2668 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:25:01.802356 2668 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:25:01.802934 2136 catalog_manager.cc:5495] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } health_report { overall_health: UNKNOWN } } }
I20260620 12:25:01.857579 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.854059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.0.98.2:41281)
0620 12:25:01.854197 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.854203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.854222 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:25:01.854298 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.854300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.854364 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.854459 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:01.854738 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.855246 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.856091 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.856320 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.856424 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.856714 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:01.857187 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:01.857299 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.857380 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.857433 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:01.857584 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.853761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41281 (local address 127.0.0.1:54590)
0620 12:25:01.853970 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.853991 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.854157 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.854480 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.854483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.854497 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:25:01.854600 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.854610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.855364 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.855368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.855975 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.855983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.856423 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.856426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.856528 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.856574 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:25:01.857274 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:25:01.857290 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.857361 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.857463 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 12:25:01.863199 2294 raft_consensus.cc:1261] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Refusing update from remote peer ceb87cd71b9140c18a608f029ad6b266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:01.863926 2666 consensus_queue.cc:1035] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:01.864055 2574 raft_consensus.cc:1261] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Refusing update from remote peer ceb87cd71b9140c18a608f029ad6b266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:01.864544 2666 consensus_queue.cc:1035] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:01.868528 2650 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: TxnStatusTablet state changed. Reason: New leader ceb87cd71b9140c18a608f029ad6b266. Latest consensus state: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.868647 2650 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:01.870461 2651 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.870579 2651 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:25:01.872575 2649 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: TxnStatusTablet state changed. Reason: New leader ceb87cd71b9140c18a608f029ad6b266. Latest consensus state: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.872689 2649 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:01.872684 2650 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.872793 2650 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P 8a22cdf4c6b94e6ea11f91828e3f113e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:01.875916 2666 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.876022 2666 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P ceb87cd71b9140c18a608f029ad6b266: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:25:01.877331 2649 tablet_replica.cc:439] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ceb87cd71b9140c18a608f029ad6b266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } } }
I20260620 12:25:01.877434 2649 tablet_replica.cc:442] T 8b9ec8df8ed341b5bdaede545710536f P 3428959204db42e483dc5678276e4959: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:01.879685 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298901204424957952 to 7298901204172992512. Current Snapshot: MvccSnapshot[applied={T|T < 7298901204424957952}]
I20260620 12:25:01.883281 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.879238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.0.98.3:35423)
0620 12:25:01.879582 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.879585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.879595 (+ 10us) server_negotiation.cc:407] Connection header received
0620 12:25:01.879672 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.879674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.879733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.879821 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:01.880173 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.880903 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.881849 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.882073 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.882176 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.882477 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:01.882902 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:01.883015 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.883099 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.883151 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260620 12:25:01.893432 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.890107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41281 (local address 127.0.98.3:40435)
0620 12:25:01.890358 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.890370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.890442 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.890691 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.890694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.890701 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:25:01.890792 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.890796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.891734 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.891736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.892308 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.892315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.892822 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.892824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.892892 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.892930 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:25:01.893225 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:25:01.893230 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.893273 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.893331 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260620 12:25:01.893520 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.890222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40435 (local address 127.0.98.2:41281)
0620 12:25:01.890363 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.890367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.890410 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:25:01.890511 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.890514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.890560 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.890638 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:01.890883 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.891644 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.892406 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.892706 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.892821 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.893060 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:01.893139 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:01.893207 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.893283 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.893395 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:25:01.900504 2294 raft_consensus.cc:1261] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:01.900539 2434 raft_consensus.cc:1261] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:01.900941 2653 consensus_queue.cc:1035] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:01.901137 2649 consensus_queue.cc:1035] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:01.905447 2671 mvcc.cc:205] Tried to move back new op lower bound from 7298901204579782656 to 7298901203173740544. Current Snapshot: MvccSnapshot[applied={T|T < 7298901204579782656}]
I20260620 12:25:01.909152 2672 mvcc.cc:205] Tried to move back new op lower bound from 7298901204579782656 to 7298901203173740544. Current Snapshot: MvccSnapshot[applied={T|T < 7298901204579782656}]
I20260620 12:25:01.924393 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.920716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60417 (local address 127.0.98.3:35423)
0620 12:25:01.920845 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:01.920849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:01.921046 (+ 197us) server_negotiation.cc:407] Connection header received
0620 12:25:01.921198 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:01.921200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:01.921240 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:01.921294 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:01.921643 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.922156 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.923327 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.923553 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.923665 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.923966 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:01.924036 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:01.924111 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:01.924196 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 12:25:01.924248 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:25:01.924419 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:01.920674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35423 (local address 127.0.98.2:60417)
0620 12:25:01.920974 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:01.920990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:01.921100 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:01.921335 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:01.921340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:01.921348 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:25:01.921543 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.921549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.922266 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.922270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.923228 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:01.923240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:01.923689 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:01.923693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:01.923788 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:01.923830 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:25:01.924115 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:25:01.924124 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:25:01.924189 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:25:01.924269 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":112,"threads_started":1}
I20260620 12:25:01.948226 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:01.948422 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:01.950901 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Sending DeleteTablet for 3 replicas of tablet 1c27978f6e4345cfb33fead3a773a591
I20260620 12:25:01.951612 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c27978f6e4345cfb33fead3a773a591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33280
I20260620 12:25:01.951738 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c27978f6e4345cfb33fead3a773a591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54584
I20260620 12:25:01.952273 2689 tablet_replica.cc:330] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: stopping tablet replica
I20260620 12:25:01.952255 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c27978f6e4345cfb33fead3a773a591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38306
I20260620 12:25:01.952636 2689 raft_consensus.cc:2229] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:01.952904 2689 raft_consensus.cc:2258] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:01.953390 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:01.953768 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:01.955503 2691 tablet_replica.cc:330] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: stopping tablet replica
I20260620 12:25:01.955847 2691 raft_consensus.cc:2229] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:01.956281 2691 raft_consensus.cc:2258] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:01.956799 2690 tablet_replica.cc:330] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: stopping tablet replica
I20260620 12:25:01.957113 2690 raft_consensus.cc:2229] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:01.958706 2691 ts_tablet_manager.cc:1905] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:01.958832 2690 raft_consensus.cc:2258] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:01.960090 2690 ts_tablet_manager.cc:1905] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:01.960531 2274 tablet_service.cc:1467] Processing CreateTablet for tablet b573c7605fa34ede9e56e393ca2e3f78 (DEFAULT_TABLE table=test-table [id=d8fe2b46e493480aa129148f04d87ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:01.960819 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b573c7605fa34ede9e56e393ca2e3f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.960826 2554 tablet_service.cc:1467] Processing CreateTablet for tablet b573c7605fa34ede9e56e393ca2e3f78 (DEFAULT_TABLE table=test-table [id=d8fe2b46e493480aa129148f04d87ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:01.961174 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b573c7605fa34ede9e56e393ca2e3f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.963608 2644 tablet_bootstrap.cc:492] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: Bootstrap starting.
I20260620 12:25:01.964401 2689 ts_tablet_manager.cc:1905] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:01.964592 2644 tablet_bootstrap.cc:654] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.965268 2643 tablet_bootstrap.cc:492] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: Bootstrap starting.
I20260620 12:25:01.966202 2643 tablet_bootstrap.cc:654] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.967825 2689 ts_tablet_manager.cc:1918] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:01.967976 2689 log.cc:1196] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/1c27978f6e4345cfb33fead3a773a591
I20260620 12:25:01.969022 2689 ts_tablet_manager.cc:1939] T 1c27978f6e4345cfb33fead3a773a591 P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting consensus metadata
I20260620 12:25:01.969700 2691 ts_tablet_manager.cc:1918] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:01.960460 2414 tablet_service.cc:1467] Processing CreateTablet for tablet b573c7605fa34ede9e56e393ca2e3f78 (DEFAULT_TABLE table=test-table [id=d8fe2b46e493480aa129148f04d87ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:01.970057 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b573c7605fa34ede9e56e393ca2e3f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:01.970333 2121 catalog_manager.cc:4841] TS 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383): tablet 1c27978f6e4345cfb33fead3a773a591 (table test-table [id=e5ce45a998664fd89ec2bbf41caf5e5a]) successfully deleted
I20260620 12:25:01.972476 2645 tablet_bootstrap.cc:492] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: Bootstrap starting.
I20260620 12:25:01.973281 2645 tablet_bootstrap.cc:654] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:01.974341 2645 tablet_bootstrap.cc:492] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: No bootstrap required, opened a new log
I20260620 12:25:01.974429 2645 ts_tablet_manager.cc:1397] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 12:25:01.974859 2645 raft_consensus.cc:348] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.974964 2645 raft_consensus.cc:374] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.974994 2645 raft_consensus.cc:729] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb87cd71b9140c18a608f029ad6b266, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.975114 2645 consensus_queue.cc:260] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.975384 2645 ts_tablet_manager.cc:1428] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:01.976907 2690 ts_tablet_manager.cc:1918] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:01.978384 2644 tablet_bootstrap.cc:492] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: No bootstrap required, opened a new log
I20260620 12:25:01.978470 2644 ts_tablet_manager.cc:1397] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260620 12:25:01.978919 2644 raft_consensus.cc:348] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.979032 2644 raft_consensus.cc:374] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.979059 2644 raft_consensus.cc:729] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a22cdf4c6b94e6ea11f91828e3f113e, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.979173 2644 consensus_queue.cc:260] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.979408 2644 ts_tablet_manager.cc:1428] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:25:01.977939 2643 tablet_bootstrap.cc:492] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: No bootstrap required, opened a new log
I20260620 12:25:01.979722 2643 ts_tablet_manager.cc:1397] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260620 12:25:01.980156 2643 raft_consensus.cc:348] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.980260 2643 raft_consensus.cc:374] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:01.980290 2643 raft_consensus.cc:729] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3428959204db42e483dc5678276e4959, State: Initialized, Role: FOLLOWER
I20260620 12:25:01.980414 2643 consensus_queue.cc:260] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:01.980679 2643 ts_tablet_manager.cc:1428] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:01.981276 2691 log.cc:1196] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/1c27978f6e4345cfb33fead3a773a591
I20260620 12:25:01.981735 2691 ts_tablet_manager.cc:1939] T 1c27978f6e4345cfb33fead3a773a591 P 3428959204db42e483dc5678276e4959: Deleting consensus metadata
I20260620 12:25:01.983088 2690 log.cc:1196] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/1c27978f6e4345cfb33fead3a773a591
I20260620 12:25:01.983119 2123 catalog_manager.cc:4841] TS 3428959204db42e483dc5678276e4959 (127.0.98.3:35423): tablet 1c27978f6e4345cfb33fead3a773a591 (table test-table [id=e5ce45a998664fd89ec2bbf41caf5e5a]) successfully deleted
I20260620 12:25:01.983465 2690 ts_tablet_manager.cc:1939] T 1c27978f6e4345cfb33fead3a773a591 P ceb87cd71b9140c18a608f029ad6b266: Deleting consensus metadata
I20260620 12:25:01.984392 2122 catalog_manager.cc:4841] TS ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281): tablet 1c27978f6e4345cfb33fead3a773a591 (table test-table [id=e5ce45a998664fd89ec2bbf41caf5e5a]) successfully deleted
I20260620 12:25:02.218868 2653 raft_consensus.cc:482] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:02.219033 2653 raft_consensus.cc:504] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:02.219556 2653 leader_election.cc:288] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:02.220028 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b573c7605fa34ede9e56e393ca2e3f78" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266" is_pre_election: true
I20260620 12:25:02.220042 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b573c7605fa34ede9e56e393ca2e3f78" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" is_pre_election: true
I20260620 12:25:02.220176 2434 raft_consensus.cc:2454] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:02.220196 2294 raft_consensus.cc:2454] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:02.220474 2506 leader_election.cc:302] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, ceb87cd71b9140c18a608f029ad6b266; no voters:
I20260620 12:25:02.220664 2653 raft_consensus.cc:2781] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:02.220718 2653 raft_consensus.cc:482] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:02.220769 2653 raft_consensus.cc:3037] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:02.221758 2653 raft_consensus.cc:504] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:02.222100 2653 leader_election.cc:288] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Requested vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:02.222434 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b573c7605fa34ede9e56e393ca2e3f78" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
I20260620 12:25:02.222455 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b573c7605fa34ede9e56e393ca2e3f78" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266"
I20260620 12:25:02.222544 2294 raft_consensus.cc:3037] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:02.222570 2434 raft_consensus.cc:3037] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:02.223446 2434 raft_consensus.cc:2454] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:02.223456 2294 raft_consensus.cc:2454] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:02.223740 2506 leader_election.cc:302] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, ceb87cd71b9140c18a608f029ad6b266; no voters:
I20260620 12:25:02.223930 2653 raft_consensus.cc:2781] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:02.224056 2653 raft_consensus.cc:686] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Becoming Leader. State: Replica: 3428959204db42e483dc5678276e4959, State: Running, Role: LEADER
I20260620 12:25:02.224202 2653 consensus_queue.cc:237] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:02.225798 2137 catalog_manager.cc:5495] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3428959204db42e483dc5678276e4959 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3428959204db42e483dc5678276e4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } health_report { overall_health: HEALTHY } } }
I20260620 12:25:02.268303 2294 raft_consensus.cc:1261] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:02.268303 2434 raft_consensus.cc:1261] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:02.268739 2649 consensus_queue.cc:1035] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:02.268896 2653 consensus_queue.cc:1035] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:25:02.282162 2675 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 12:25:03.841136 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:03.841322 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:03.843765 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Sending DeleteTablet for 3 replicas of tablet b573c7605fa34ede9e56e393ca2e3f78
I20260620 12:25:03.844345 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet b573c7605fa34ede9e56e393ca2e3f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54584
I20260620 12:25:03.844411 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet b573c7605fa34ede9e56e393ca2e3f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38306
I20260620 12:25:03.844405 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet b573c7605fa34ede9e56e393ca2e3f78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33280
I20260620 12:25:03.844728 2721 tablet_replica.cc:330] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: stopping tablet replica
I20260620 12:25:03.844833 2722 tablet_replica.cc:330] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: stopping tablet replica
I20260620 12:25:03.844907 2721 raft_consensus.cc:2229] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:03.844976 2722 raft_consensus.cc:2229] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:03.845052 2721 raft_consensus.cc:2258] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:03.845206 2722 raft_consensus.cc:2258] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:03.845870 2721 ts_tablet_manager.cc:1905] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:03.845907 2722 ts_tablet_manager.cc:1905] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:03.847872 2723 tablet_replica.cc:330] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: stopping tablet replica
I20260620 12:25:03.848079 2723 raft_consensus.cc:2229] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:03.848214 2723 raft_consensus.cc:2258] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:03.848847 2723 ts_tablet_manager.cc:1905] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:03.850713 2722 ts_tablet_manager.cc:1918] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:25:03.850731 2721 ts_tablet_manager.cc:1918] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:25:03.850821 2721 log.cc:1196] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/b573c7605fa34ede9e56e393ca2e3f78
I20260620 12:25:03.850869 2722 log.cc:1196] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/b573c7605fa34ede9e56e393ca2e3f78
I20260620 12:25:03.851163 2723 ts_tablet_manager.cc:1918] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:25:03.851214 2722 ts_tablet_manager.cc:1939] T b573c7605fa34ede9e56e393ca2e3f78 P 3428959204db42e483dc5678276e4959: Deleting consensus metadata
I20260620 12:25:03.851215 2721 ts_tablet_manager.cc:1939] T b573c7605fa34ede9e56e393ca2e3f78 P ceb87cd71b9140c18a608f029ad6b266: Deleting consensus metadata
I20260620 12:25:03.851274 2723 log.cc:1196] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/b573c7605fa34ede9e56e393ca2e3f78
I20260620 12:25:03.851550 2723 ts_tablet_manager.cc:1939] T b573c7605fa34ede9e56e393ca2e3f78 P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting consensus metadata
I20260620 12:25:03.852102 2123 catalog_manager.cc:4841] TS 3428959204db42e483dc5678276e4959 (127.0.98.3:35423): tablet b573c7605fa34ede9e56e393ca2e3f78 (table test-table [id=d8fe2b46e493480aa129148f04d87ce1]) successfully deleted
I20260620 12:25:03.852275 2122 catalog_manager.cc:4841] TS ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281): tablet b573c7605fa34ede9e56e393ca2e3f78 (table test-table [id=d8fe2b46e493480aa129148f04d87ce1]) successfully deleted
I20260620 12:25:03.852326 2121 catalog_manager.cc:4841] TS 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383): tablet b573c7605fa34ede9e56e393ca2e3f78 (table test-table [id=d8fe2b46e493480aa129148f04d87ce1]) successfully deleted
Jun 20 12:25:03 dist-test-slave-bwpp krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:25:03 dist-test-slave-bwpp krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:03.874078 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:03.864922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.98.62:40507)
0620 12:25:03.865197 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:03.865208 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:03.865341 (+ 133us) server_negotiation.cc:407] Connection header received
0620 12:25:03.865476 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:03.865480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:03.865528 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:03.865630 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:03.866403 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:03.867051 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:03.867920 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:03.868148 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:03.868218 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:03.870828 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:03.870831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:03.870833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:03.870855 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:03.872472 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:03.872879 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:03.872882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:03.872883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:03.872967 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:03.873209 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:03.873211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:03.873213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:03.873416 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:03.873521 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:03.873728 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 12:25:03.873835 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
I20260620 12:25:03.876878 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:03.877337 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:03.882141 2414 tablet_service.cc:1467] Processing CreateTablet for tablet e07bcb24ab934adfbe2519f91292961b (DEFAULT_TABLE table=test-table [id=601cd2b0543542d5b61a3a87a7bf367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:03.882128 2274 tablet_service.cc:1467] Processing CreateTablet for tablet e07bcb24ab934adfbe2519f91292961b (DEFAULT_TABLE table=test-table [id=601cd2b0543542d5b61a3a87a7bf367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:03.882205 2554 tablet_service.cc:1467] Processing CreateTablet for tablet e07bcb24ab934adfbe2519f91292961b (DEFAULT_TABLE table=test-table [id=601cd2b0543542d5b61a3a87a7bf367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:03.882457 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07bcb24ab934adfbe2519f91292961b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:03.882457 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07bcb24ab934adfbe2519f91292961b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:03.882457 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07bcb24ab934adfbe2519f91292961b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:03.884511 2732 tablet_bootstrap.cc:492] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: Bootstrap starting.
I20260620 12:25:03.885099 2733 tablet_bootstrap.cc:492] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: Bootstrap starting.
I20260620 12:25:03.885191 2734 tablet_bootstrap.cc:492] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: Bootstrap starting.
I20260620 12:25:03.885550 2732 tablet_bootstrap.cc:654] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:03.886179 2734 tablet_bootstrap.cc:654] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:03.886507 2733 tablet_bootstrap.cc:654] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:03.886819 2732 tablet_bootstrap.cc:492] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: No bootstrap required, opened a new log
I20260620 12:25:03.886957 2732 ts_tablet_manager.cc:1397] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:03.887380 2734 tablet_bootstrap.cc:492] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: No bootstrap required, opened a new log
I20260620 12:25:03.887475 2734 ts_tablet_manager.cc:1397] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 12:25:03.887471 2732 raft_consensus.cc:348] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:03.887621 2732 raft_consensus.cc:374] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:03.887686 2732 raft_consensus.cc:729] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb87cd71b9140c18a608f029ad6b266, State: Initialized, Role: FOLLOWER
I20260620 12:25:03.887823 2732 consensus_queue.cc:260] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:03.887893 2734 raft_consensus.cc:348] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:03.888016 2734 raft_consensus.cc:374] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:03.888048 2734 raft_consensus.cc:729] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3428959204db42e483dc5678276e4959, State: Initialized, Role: FOLLOWER
I20260620 12:25:03.888108 2732 ts_tablet_manager.cc:1428] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:03.888162 2734 consensus_queue.cc:260] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:03.888813 2734 ts_tablet_manager.cc:1428] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:25:03.889099 2733 tablet_bootstrap.cc:492] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: No bootstrap required, opened a new log
I20260620 12:25:03.889189 2733 ts_tablet_manager.cc:1397] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:25:03.890054 2733 raft_consensus.cc:348] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:03.890234 2733 raft_consensus.cc:374] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:03.890318 2733 raft_consensus.cc:729] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a22cdf4c6b94e6ea11f91828e3f113e, State: Initialized, Role: FOLLOWER
I20260620 12:25:03.890515 2733 consensus_queue.cc:260] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:03.891435 2733 ts_tablet_manager.cc:1428] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:04.251328 2712 raft_consensus.cc:482] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:04.251463 2712 raft_consensus.cc:504] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.251927 2712 leader_election.cc:288] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:04.252426 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07bcb24ab934adfbe2519f91292961b" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266" is_pre_election: true
I20260620 12:25:04.252447 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07bcb24ab934adfbe2519f91292961b" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" is_pre_election: true
I20260620 12:25:04.252575 2434 raft_consensus.cc:2454] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:04.252624 2294 raft_consensus.cc:2454] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:04.252969 2506 leader_election.cc:302] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, ceb87cd71b9140c18a608f029ad6b266; no voters:
I20260620 12:25:04.253158 2712 raft_consensus.cc:2781] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:04.253216 2712 raft_consensus.cc:482] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:04.253243 2712 raft_consensus.cc:3037] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:04.254223 2712 raft_consensus.cc:504] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.254549 2712 leader_election.cc:288] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Requested vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:04.254889 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07bcb24ab934adfbe2519f91292961b" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
I20260620 12:25:04.255023 2294 raft_consensus.cc:3037] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:04.254989 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e07bcb24ab934adfbe2519f91292961b" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266"
I20260620 12:25:04.255154 2434 raft_consensus.cc:3037] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:04.255952 2294 raft_consensus.cc:2454] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:04.256040 2434 raft_consensus.cc:2454] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:04.256306 2505 leader_election.cc:302] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, 8a22cdf4c6b94e6ea11f91828e3f113e; no voters:
I20260620 12:25:04.256480 2712 raft_consensus.cc:2781] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:04.256603 2712 raft_consensus.cc:686] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Becoming Leader. State: Replica: 3428959204db42e483dc5678276e4959, State: Running, Role: LEADER
I20260620 12:25:04.256753 2712 consensus_queue.cc:237] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.258543 2137 catalog_manager.cc:5495] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3428959204db42e483dc5678276e4959 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3428959204db42e483dc5678276e4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } health_report { overall_health: HEALTHY } } }
I20260620 12:25:04.346123 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:04.342953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.0.98.3:35423)
0620 12:25:04.343145 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:04.343158 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:04.343173 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:04.343226 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:04.343229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:04.343280 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:04.343365 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:04.343632 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:04.344157 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:04.345010 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:04.345246 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:04.345354 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:04.345635 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:04.345707 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:04.345788 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:04.345910 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 12:25:04.345960 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":49,"threads_started":1}
I20260620 12:25:04.351895 2434 raft_consensus.cc:1261] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:04.351924 2294 raft_consensus.cc:1261] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:04.352324 2699 consensus_queue.cc:1035] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:04.352687 2699 consensus_queue.cc:1035] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:04.382238 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:04.382411 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:04.384289 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Sending DeleteTablet for 3 replicas of tablet e07bcb24ab934adfbe2519f91292961b
I20260620 12:25:04.384824 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet e07bcb24ab934adfbe2519f91292961b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33280
I20260620 12:25:04.384858 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet e07bcb24ab934adfbe2519f91292961b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38306
I20260620 12:25:04.385150 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet e07bcb24ab934adfbe2519f91292961b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54584
I20260620 12:25:04.385187 2755 tablet_replica.cc:330] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: stopping tablet replica
I20260620 12:25:04.385321 2755 raft_consensus.cc:2229] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:04.385485 2755 raft_consensus.cc:2258] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:04.385695 2757 tablet_replica.cc:330] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: stopping tablet replica
I20260620 12:25:04.385733 2756 tablet_replica.cc:330] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: stopping tablet replica
I20260620 12:25:04.385735 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:25:04.385896 2757 raft_consensus.cc:2229] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:04.385918 2756 raft_consensus.cc:2229] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:04.386026 2757 raft_consensus.cc:2258] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:04.386089 2756 raft_consensus.cc:2258] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 12:25:04.386111 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:04.386341 2755 ts_tablet_manager.cc:1905] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:04.386729 2757 ts_tablet_manager.cc:1905] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:04.387729 2756 ts_tablet_manager.cc:1905] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:04.388948 2755 ts_tablet_manager.cc:1918] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:04.389199 2755 log.cc:1196] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/e07bcb24ab934adfbe2519f91292961b
I20260620 12:25:04.389372 2757 ts_tablet_manager.cc:1918] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:04.389456 2757 log.cc:1196] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/e07bcb24ab934adfbe2519f91292961b
I20260620 12:25:04.389572 2755 ts_tablet_manager.cc:1939] T e07bcb24ab934adfbe2519f91292961b P 8a22cdf4c6b94e6ea11f91828e3f113e: Deleting consensus metadata
I20260620 12:25:04.389721 2757 ts_tablet_manager.cc:1939] T e07bcb24ab934adfbe2519f91292961b P ceb87cd71b9140c18a608f029ad6b266: Deleting consensus metadata
I20260620 12:25:04.390244 2121 catalog_manager.cc:4841] TS 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383): tablet e07bcb24ab934adfbe2519f91292961b (table test-table [id=601cd2b0543542d5b61a3a87a7bf367d]) successfully deleted
I20260620 12:25:04.390291 2122 catalog_manager.cc:4841] TS ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281): tablet e07bcb24ab934adfbe2519f91292961b (table test-table [id=601cd2b0543542d5b61a3a87a7bf367d]) successfully deleted
I20260620 12:25:04.390810 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 49021eb07cd8434e828c0b0886e18326 (DEFAULT_TABLE table=test-table [id=b0eb3f429ee44798ac859f958a197a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:04.390810 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 49021eb07cd8434e828c0b0886e18326 (DEFAULT_TABLE table=test-table [id=b0eb3f429ee44798ac859f958a197a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:04.390928 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 49021eb07cd8434e828c0b0886e18326 (DEFAULT_TABLE table=test-table [id=b0eb3f429ee44798ac859f958a197a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:04.391058 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49021eb07cd8434e828c0b0886e18326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:04.391058 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49021eb07cd8434e828c0b0886e18326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:04.391178 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49021eb07cd8434e828c0b0886e18326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:04.392318 2756 ts_tablet_manager.cc:1918] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:04.392383 2756 log.cc:1196] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/e07bcb24ab934adfbe2519f91292961b
I20260620 12:25:04.392657 2756 ts_tablet_manager.cc:1939] T e07bcb24ab934adfbe2519f91292961b P 3428959204db42e483dc5678276e4959: Deleting consensus metadata
I20260620 12:25:04.393257 2758 tablet_bootstrap.cc:492] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: Bootstrap starting.
I20260620 12:25:04.393445 2759 tablet_bootstrap.cc:492] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: Bootstrap starting.
I20260620 12:25:04.393466 2733 tablet_bootstrap.cc:492] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e: Bootstrap starting.
I20260620 12:25:04.394008 2759 tablet_bootstrap.cc:654] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:04.394052 2758 tablet_bootstrap.cc:654] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:04.394301 2733 tablet_bootstrap.cc:654] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:04.394984 2759 tablet_bootstrap.cc:492] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: No bootstrap required, opened a new log
I20260620 12:25:04.395071 2759 ts_tablet_manager.cc:1397] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:04.395098 2123 catalog_manager.cc:4841] TS 3428959204db42e483dc5678276e4959 (127.0.98.3:35423): tablet e07bcb24ab934adfbe2519f91292961b (table test-table [id=601cd2b0543542d5b61a3a87a7bf367d]) successfully deleted
I20260620 12:25:04.395412 2759 raft_consensus.cc:348] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.395532 2759 raft_consensus.cc:374] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:04.395576 2759 raft_consensus.cc:729] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3428959204db42e483dc5678276e4959, State: Initialized, Role: FOLLOWER
I20260620 12:25:04.395597 2733 tablet_bootstrap.cc:492] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e: No bootstrap required, opened a new log
I20260620 12:25:04.395669 2733 ts_tablet_manager.cc:1397] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:04.395704 2759 consensus_queue.cc:260] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.395970 2759 ts_tablet_manager.cc:1428] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:25:04.396082 2733 raft_consensus.cc:348] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.396196 2733 raft_consensus.cc:374] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:04.396234 2733 raft_consensus.cc:729] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a22cdf4c6b94e6ea11f91828e3f113e, State: Initialized, Role: FOLLOWER
I20260620 12:25:04.396370 2733 consensus_queue.cc:260] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.396656 2733 ts_tablet_manager.cc:1428] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:25:04.396920 2758 tablet_bootstrap.cc:492] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: No bootstrap required, opened a new log
I20260620 12:25:04.396998 2758 ts_tablet_manager.cc:1397] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 12:25:04.397389 2758 raft_consensus.cc:348] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.397503 2758 raft_consensus.cc:374] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:04.397537 2758 raft_consensus.cc:729] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb87cd71b9140c18a608f029ad6b266, State: Initialized, Role: FOLLOWER
I20260620 12:25:04.397680 2758 consensus_queue.cc:260] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.397935 2758 ts_tablet_manager.cc:1428] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:04.557806 2712 raft_consensus.cc:482] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:04.558015 2712 raft_consensus.cc:504] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.558537 2712 leader_election.cc:288] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:04.559026 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49021eb07cd8434e828c0b0886e18326" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" is_pre_election: true
I20260620 12:25:04.559044 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49021eb07cd8434e828c0b0886e18326" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266" is_pre_election: true
I20260620 12:25:04.559188 2294 raft_consensus.cc:2454] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:04.559222 2434 raft_consensus.cc:2454] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 0.
I20260620 12:25:04.559531 2505 leader_election.cc:302] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, 8a22cdf4c6b94e6ea11f91828e3f113e; no voters:
I20260620 12:25:04.559715 2712 raft_consensus.cc:2781] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:04.559772 2712 raft_consensus.cc:482] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:04.559813 2712 raft_consensus.cc:3037] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:04.560772 2712 raft_consensus.cc:504] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.561115 2712 leader_election.cc:288] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Requested vote from peers 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383), ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281)
I20260620 12:25:04.561441 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49021eb07cd8434e828c0b0886e18326" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e"
I20260620 12:25:04.561499 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49021eb07cd8434e828c0b0886e18326" candidate_uuid: "3428959204db42e483dc5678276e4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb87cd71b9140c18a608f029ad6b266"
I20260620 12:25:04.561556 2294 raft_consensus.cc:3037] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:04.561643 2434 raft_consensus.cc:3037] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:04.562464 2294 raft_consensus.cc:2454] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:04.562641 2434 raft_consensus.cc:2454] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3428959204db42e483dc5678276e4959 in term 1.
I20260620 12:25:04.562783 2505 leader_election.cc:302] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3428959204db42e483dc5678276e4959, 8a22cdf4c6b94e6ea11f91828e3f113e; no voters:
I20260620 12:25:04.562956 2712 raft_consensus.cc:2781] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:04.563071 2712 raft_consensus.cc:686] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Becoming Leader. State: Replica: 3428959204db42e483dc5678276e4959, State: Running, Role: LEADER
I20260620 12:25:04.563208 2712 consensus_queue.cc:237] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } }
I20260620 12:25:04.564920 2136 catalog_manager.cc:5495] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3428959204db42e483dc5678276e4959 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3428959204db42e483dc5678276e4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3428959204db42e483dc5678276e4959" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35423 } health_report { overall_health: HEALTHY } } }
I20260620 12:25:04.596994 2294 raft_consensus.cc:1261] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:04.596940 2434 raft_consensus.cc:1261] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Refusing update from remote peer 3428959204db42e483dc5678276e4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:04.597330 2712 consensus_queue.cc:1035] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb87cd71b9140c18a608f029ad6b266" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:04.597462 2754 consensus_queue.cc:1035] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a22cdf4c6b94e6ea11f91828e3f113e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:25:04.611248 2742 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 12:25:06.173916 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:06.174057 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:06.176337 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc8336a847774b2d89f71730029d88f0: Sending DeleteTablet for 3 replicas of tablet 49021eb07cd8434e828c0b0886e18326
I20260620 12:25:06.176947 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 49021eb07cd8434e828c0b0886e18326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33280
I20260620 12:25:06.176952 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 49021eb07cd8434e828c0b0886e18326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54584
I20260620 12:25:06.177001 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 49021eb07cd8434e828c0b0886e18326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38306
I20260620 12:25:06.177316 2786 tablet_replica.cc:330] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: stopping tablet replica
I20260620 12:25:06.177338 2787 tablet_replica.cc:330] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: stopping tablet replica
I20260620 12:25:06.177443 2788 tablet_replica.cc:330] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e: stopping tablet replica
I20260620 12:25:06.177462 2787 raft_consensus.cc:2229] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:06.177495 2786 raft_consensus.cc:2229] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:06.177620 2786 raft_consensus.cc:2258] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:06.177639 2787 raft_consensus.cc:2258] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:06.177660 2788 raft_consensus.cc:2229] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:06.177798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 2209
I20260620 12:25:06.178030 2788 raft_consensus.cc:2258] T 49021eb07cd8434e828c0b0886e18326 P 8a22cdf4c6b94e6ea11f91828e3f113e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:06.178815 2786 ts_tablet_manager.cc:1905] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:06.179031 2787 ts_tablet_manager.cc:1905] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:06.181491 2786 ts_tablet_manager.cc:1918] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:25:06.181608 2786 log.cc:1196] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/49021eb07cd8434e828c0b0886e18326
I20260620 12:25:06.181726 2787 ts_tablet_manager.cc:1918] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:25:06.181818 2787 log.cc:1196] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/49021eb07cd8434e828c0b0886e18326
I20260620 12:25:06.181912 2786 ts_tablet_manager.cc:1939] T 49021eb07cd8434e828c0b0886e18326 P ceb87cd71b9140c18a608f029ad6b266: Deleting consensus metadata
I20260620 12:25:06.182126 2787 ts_tablet_manager.cc:1939] T 49021eb07cd8434e828c0b0886e18326 P 3428959204db42e483dc5678276e4959: Deleting consensus metadata
I20260620 12:25:06.182735 2123 catalog_manager.cc:4841] TS 3428959204db42e483dc5678276e4959 (127.0.98.3:35423): tablet 49021eb07cd8434e828c0b0886e18326 (table test-table [id=b0eb3f429ee44798ac859f958a197a4f]) successfully deleted
I20260620 12:25:06.182737 2122 catalog_manager.cc:4841] TS ceb87cd71b9140c18a608f029ad6b266 (127.0.98.2:41281): tablet 49021eb07cd8434e828c0b0886e18326 (table test-table [id=b0eb3f429ee44798ac859f958a197a4f]) successfully deleted
I20260620 12:25:06.184994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 2349
W20260620 12:25:06.185015 2121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36383 (error 108)
I20260620 12:25:06.189790 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:06.189363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36383 (local address 127.0.0.1:33286)
0620 12:25:06.189637 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:06.189705 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
W20260620 12:25:06.189960 2121 catalog_manager.cc:4584] TS 8a22cdf4c6b94e6ea11f91828e3f113e (127.0.98.1:36383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49021eb07cd8434e828c0b0886e18326: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36383: connect: Connection refused (error 111)
I20260620 12:25:06.192036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 2489
I20260620 12:25:06.198459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 2105
2026-06-20T12:25:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6906 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:06 dist-test-slave-bwpp krb5kdc[2793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:06 dist-test-slave-bwpp krb5kdc[2793](info): set up 2 sockets
Jun 20 12:25:06 dist-test-slave-bwpp krb5kdc[2793](info): commencing operation
krb5kdc: starting...
W20260620 12:25:07.250713 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:07 dist-test-slave-bwpp krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:07.367460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42885
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:07.473150 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:07.473439 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:07.473520 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:07.477437 2806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:07.477526 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:07.477566 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:07.477624 2806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:07.477665 2806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:07.482735 2806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:07.483739 2806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:07.484743 2806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:07.493046 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:07.493046 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:07.493256 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:07.493114 2806 server_base.cc:1034] running on GCE node
I20260620 12:25:07.493913 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:07.495048 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:07.496228 2806 hybrid_clock.cc:648] HybridClock initialized: now 1781958307496202 us; error 39 us; skew 500 ppm
Jun 20 12:25:07 dist-test-slave-bwpp krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:07.498876 2806 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:07.499666 2806 webserver.cc:460] Webserver started at http://127.0.98.62:40583/ using document root <none> and password file <none>
I20260620 12:25:07.500202 2806 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:07.500290 2806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:07.500548 2806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:07.502530 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "6e971d6339ec41148b4dbd28feebac4f"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "34b408e0bed01c0beaa2bca1f9b0dfac"
server_key_iv: "3f14e144634f56722e5284e2e033a939"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.503125 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e971d6339ec41148b4dbd28feebac4f"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "34b408e0bed01c0beaa2bca1f9b0dfac"
server_key_iv: "3f14e144634f56722e5284e2e033a939"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.506594 2806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:07.508808 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:07.509989 2806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:25:07.510138 2806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "6e971d6339ec41148b4dbd28feebac4f"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "34b408e0bed01c0beaa2bca1f9b0dfac"
server_key_iv: "3f14e144634f56722e5284e2e033a939"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.510239 2806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:07.524211 2806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:07.527046 2806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:07.527241 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:07.534109 2806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42885
I20260620 12:25:07.534214 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42885 every 8 connection(s)
I20260620 12:25:07.535130 2806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:07.537804 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:07.542922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 2806
I20260620 12:25:07.543011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:07.543238 392 external_mini_cluster.cc:1428] Setting key 1e9e22ca94fa3621c088968bd39af586
I20260620 12:25:07.543258 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f: Bootstrap starting.
I20260620 12:25:07.545897 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:07.546779 2874 log.cc:824] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:07.548575 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f: No bootstrap required, opened a new log
Jun 20 12:25:07 dist-test-slave-bwpp krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:07.552217 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e971d6339ec41148b4dbd28feebac4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42885 } }
I20260620 12:25:07.552515 2874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:07.552593 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e971d6339ec41148b4dbd28feebac4f, State: Initialized, Role: FOLLOWER
I20260620 12:25:07.553102 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e971d6339ec41148b4dbd28feebac4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42885 } }
I20260620 12:25:07.553265 2874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:07.553330 2874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:07.553480 2874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:07.554668 2874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e971d6339ec41148b4dbd28feebac4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42885 } }
I20260620 12:25:07.555038 2874 leader_election.cc:302] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e971d6339ec41148b4dbd28feebac4f; no voters:
I20260620 12:25:07.555404 2874 leader_election.cc:288] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:07.555469 2879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:07.555688 2879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [term 1 LEADER]: Becoming Leader. State: Replica: 6e971d6339ec41148b4dbd28feebac4f, State: Running, Role: LEADER
I20260620 12:25:07.556031 2879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e971d6339ec41148b4dbd28feebac4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42885 } }
I20260620 12:25:07.556710 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:07.557374 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e971d6339ec41148b4dbd28feebac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e971d6339ec41148b4dbd28feebac4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42885 } } }
I20260620 12:25:07.557540 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:07.557792 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e971d6339ec41148b4dbd28feebac4f. Latest consensus state: current_term: 1 leader_uuid: "6e971d6339ec41148b4dbd28feebac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e971d6339ec41148b4dbd28feebac4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42885 } } }
I20260620 12:25:07.557911 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:07.558589 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:07.544555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.0.98.62:42885)
0620 12:25:07.544972 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:07.544998 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:07.545040 (+ 42us) server_negotiation.cc:407] Connection header received
0620 12:25:07.545830 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:07.545855 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:07.546295 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:07.546676 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:07.547549 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:07.548507 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:07.549394 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:07.550104 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:07.550264 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:07.552854 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:07.552860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:07.552876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:07.552906 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:07.554986 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:07.555509 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:07.555515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:07.555520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:07.555681 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:07.555986 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:07.555989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:07.555991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:07.556409 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:07.556634 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:07.556906 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 12:25:07.557142 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":105,"threads_started":1}
I20260620 12:25:07.559813 2884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:07.562206 2884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:07.566944 2884 catalog_manager.cc:1347] Generated new cluster ID: 31124bce2da24d60b16b94dd1a79ab6d
I20260620 12:25:07.567039 2884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:07.584777 2884 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:07.585767 2884 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:07.591508 2884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e971d6339ec41148b4dbd28feebac4f: Generated new TSK 0
I20260620 12:25:07.592149 2884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:25:07.623057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42885
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:07.732328 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:07.732592 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:07.732666 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:07.737075 2900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:07.737154 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:07.737252 2900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:07.742468 2900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:07.743485 2900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:07.744453 2900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:07.754356 2905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:07.754406 2906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:07.754420 2908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:07.754640 2900 server_base.cc:1034] running on GCE node
I20260620 12:25:07.755120 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:07.755738 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:07.756903 2900 hybrid_clock.cc:648] HybridClock initialized: now 1781958307756892 us; error 24 us; skew 500 ppm
Jun 20 12:25:07 dist-test-slave-bwpp krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:07.759835 2900 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:07.760654 2900 webserver.cc:460] Webserver started at http://127.0.98.1:36061/ using document root <none> and password file <none>
I20260620 12:25:07.761209 2900 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:07.761294 2900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:07.761523 2900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:07.763554 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "990889da12ca4aaea08580a44471192b"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "343dfd32bbbcb8ce41cb8f52968887c4"
server_key_iv: "5ad236a92134bcbcf5859baa1ed0511e"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.764111 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "990889da12ca4aaea08580a44471192b"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "343dfd32bbbcb8ce41cb8f52968887c4"
server_key_iv: "5ad236a92134bcbcf5859baa1ed0511e"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.767661 2900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:25:07.769959 2915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:07.771257 2900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:07.771387 2900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "990889da12ca4aaea08580a44471192b"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "343dfd32bbbcb8ce41cb8f52968887c4"
server_key_iv: "5ad236a92134bcbcf5859baa1ed0511e"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.771499 2900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:07.797194 2900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:07.800814 2900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:07.801024 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:07.801625 2900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:07.802474 2900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:07.802553 2900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:07.802619 2900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:07.802675 2900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:07.811632 2900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45029
I20260620 12:25:07.811650 3028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45029 every 8 connection(s)
I20260620 12:25:07.812521 2900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:07.819449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 2900
I20260620 12:25:07.819552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:07.819818 392 external_mini_cluster.cc:1428] Setting key 1e17d718919692e46be1a578bca2adee
Jun 20 12:25:07 dist-test-slave-bwpp krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:07.827404 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:07.814366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45163 (local address 127.0.98.62:42885)
0620 12:25:07.814508 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:07.814512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:07.815425 (+ 913us) server_negotiation.cc:407] Connection header received
0620 12:25:07.816799 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:07.816802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:07.816862 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:07.816934 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:07.818028 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:07.818582 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:07.820075 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:07.820309 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:07.820430 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:07.823438 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:07.823441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:07.823443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:07.823467 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:07.824947 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:07.825579 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:07.825581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:07.825583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:07.825656 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:07.826041 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:07.826047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:07.826052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:07.826329 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:07.826446 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:07.827024 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 12:25:07.827174 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:25:07.828050 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:07.814789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42885 (local address 127.0.98.1:45163)
0620 12:25:07.815266 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:07.815316 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:07.816602 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:07.817061 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:07.817068 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:07.817417 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:07.817893 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:07.817907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:07.818751 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:07.818754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:07.819932 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:07.819943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:07.820467 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:07.820471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:07.820659 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:07.821342 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:07.821364 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:07.823250 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:07.825065 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:07.825072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:07.825089 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:07.825468 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:07.825749 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:07.825753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:07.825756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:07.825922 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:07.826441 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:07.826447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:07.826724 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 12:25:07.826922 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 12:25:07.827174 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":109,"threads_started":1}
I20260620 12:25:07.829280 3029 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42885
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:25:07.829521 3029 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:07.829994 3029 heartbeater.cc:511] Master 127.0.98.62:42885 requested a full tablet report, sending...
I20260620 12:25:07.831514 2838 ts_manager.cc:194] Registered new tserver with Master: 990889da12ca4aaea08580a44471192b (127.0.98.1:45029)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:07.832587 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:07.846060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42885
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:07.955761 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:07.956020 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:07.956107 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:07.960340 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:07.960402 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:07.960513 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:07.965763 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:07.966825 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:07.967850 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:07.977771 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:07.977705 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:07.977742 3034 server_base.cc:1034] running on GCE node
W20260620 12:25:07.977705 3042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:07.978348 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:07.979056 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:07.980244 3034 hybrid_clock.cc:648] HybridClock initialized: now 1781958307980206 us; error 49 us; skew 500 ppm
Jun 20 12:25:07 dist-test-slave-bwpp krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:07.983101 3034 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:07.983928 3034 webserver.cc:460] Webserver started at http://127.0.98.2:45731/ using document root <none> and password file <none>
I20260620 12:25:07.984546 3034 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:07.984607 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:07.984766 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:07.986718 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5fc7ab116354c458d73285480fb468d"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "28ed8ed3d3c00e3621af8a5fce0f8b4d"
server_key_iv: "cb98f897bde9a7d96eabbb7cd67790e1"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.987218 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5fc7ab116354c458d73285480fb468d"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "28ed8ed3d3c00e3621af8a5fce0f8b4d"
server_key_iv: "cb98f897bde9a7d96eabbb7cd67790e1"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.990639 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:07.992904 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:07.994053 3034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:25:07.994187 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "f5fc7ab116354c458d73285480fb468d"
format_stamp: "Formatted at 2026-06-20 12:25:07 on dist-test-slave-bwpp"
server_key: "28ed8ed3d3c00e3621af8a5fce0f8b4d"
server_key_iv: "cb98f897bde9a7d96eabbb7cd67790e1"
server_key_version: "encryptionkey@0"
I20260620 12:25:07.994292 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:08.004109 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:08.007231 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:08.007428 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:08.007982 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:08.008736 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:08.008806 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:08.008886 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:08.008941 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:08.017608 3034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43433
I20260620 12:25:08.017640 3162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43433 every 8 connection(s)
I20260620 12:25:08.018455 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:08.021836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3034
I20260620 12:25:08.021955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:08.022225 392 external_mini_cluster.cc:1428] Setting key 02c7a4f9f9ea241c0b85a075e425a167
Jun 20 12:25:08 dist-test-slave-bwpp krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:25:08.032617 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:08.020221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49409 (local address 127.0.98.62:42885)
0620 12:25:08.020374 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:08.020379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:08.021210 (+ 831us) server_negotiation.cc:407] Connection header received
0620 12:25:08.022732 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:08.022736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:08.022796 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:08.022907 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:08.024173 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.024677 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.025983 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.026393 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.026513 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:08.028892 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:08.028899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:08.028905 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:08.028938 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:08.030718 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:08.031189 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:08.031196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:08.031200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:08.031289 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:08.031589 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:08.031594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:08.031599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:08.031865 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:08.031980 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:08.032394 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 12:25:08.032497 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:25:08.033452 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:08.020608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42885 (local address 127.0.98.2:49409)
0620 12:25:08.021052 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:08.021103 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:08.022516 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:08.023075 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:08.023082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:08.023413 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:08.024021 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:08.024041 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.024802 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.024809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:08.025857 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:08.025866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.026533 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.026536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:08.026725 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:08.027320 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:08.027338 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:08.028728 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:08.030829 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:08.030834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:08.030846 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:08.031081 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:08.031382 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:08.031385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:08.031386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:08.031500 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:08.031970 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:08.031974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:08.032189 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 12:25:08.032360 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 12:25:08.032594 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:25:08.034680 3163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42885
I20260620 12:25:08.034933 3163 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:08.035485 3163 heartbeater.cc:511] Master 127.0.98.62:42885 requested a full tablet report, sending...
I20260620 12:25:08.036672 2838 ts_manager.cc:194] Registered new tserver with Master: f5fc7ab116354c458d73285480fb468d (127.0.98.2:43433)
I20260620 12:25:08.037535 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:08.049602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42885
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:08.161679 3168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:08.161921 3168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:08.161962 3168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:08.166157 3168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:08.166208 3168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:08.166299 3168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:08.171476 3168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:08.172437 3168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:08.173450 3168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:08.183065 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:08.183076 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:08.183079 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:08.183529 3168 server_base.cc:1034] running on GCE node
I20260620 12:25:08.183944 3168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:08.184692 3168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:08.185871 3168 hybrid_clock.cc:648] HybridClock initialized: now 1781958308185835 us; error 52 us; skew 500 ppm
Jun 20 12:25:08 dist-test-slave-bwpp krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:08.188535 3168 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:08.189400 3168 webserver.cc:460] Webserver started at http://127.0.98.3:46467/ using document root <none> and password file <none>
I20260620 12:25:08.189955 3168 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:08.190053 3168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:08.190279 3168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:08.192294 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf3e5d19586743d384872783f64ca83f"
format_stamp: "Formatted at 2026-06-20 12:25:08 on dist-test-slave-bwpp"
server_key: "49b7756975b2538aab133c1d7e384a96"
server_key_iv: "a6d86ed1df3c9edae536a3dd858f58b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:08.192848 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf3e5d19586743d384872783f64ca83f"
format_stamp: "Formatted at 2026-06-20 12:25:08 on dist-test-slave-bwpp"
server_key: "49b7756975b2538aab133c1d7e384a96"
server_key_iv: "a6d86ed1df3c9edae536a3dd858f58b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:08.196455 3168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 12:25:08.198856 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:08.200018 3168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:25:08.200142 3168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "bf3e5d19586743d384872783f64ca83f"
format_stamp: "Formatted at 2026-06-20 12:25:08 on dist-test-slave-bwpp"
server_key: "49b7756975b2538aab133c1d7e384a96"
server_key_iv: "a6d86ed1df3c9edae536a3dd858f58b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:08.200246 3168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:08.264215 3168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:08.267091 3168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:08.267297 3168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:08.267887 3168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:08.268707 3168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:08.268785 3168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:08.268879 3168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:08.268932 3168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:08.278030 3168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38393
I20260620 12:25:08.278057 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38393 every 8 connection(s)
I20260620 12:25:08.278959 3168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:08.285907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3168
I20260620 12:25:08.286034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:08.286350 392 external_mini_cluster.cc:1428] Setting key 639d5f435f9879a081391637541260bc
Jun 20 12:25:08 dist-test-slave-bwpp krb5kdc[2793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:08.292886 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:08.280839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46701 (local address 127.0.98.62:42885)
0620 12:25:08.280978 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:08.280981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:08.281823 (+ 842us) server_negotiation.cc:407] Connection header received
0620 12:25:08.283261 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:08.283265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:08.283322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:08.283428 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:08.284570 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.285107 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.286367 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.286686 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.286797 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:08.289651 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:08.289655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:08.289657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:08.289680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:08.291132 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:08.291601 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:08.291604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:08.291605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:08.291677 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:08.291975 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:08.291977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:08.291978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:08.292164 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:08.292239 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:08.292684 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 12:25:08.292772 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:25:08.293701 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:08.281234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42885 (local address 127.0.98.3:46701)
0620 12:25:08.281680 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:08.281721 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:08.283050 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:08.283605 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:08.283611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:08.283961 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:08.284415 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:08.284428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.285243 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.285246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:08.286245 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:08.286255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:08.286891 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:08.286895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:08.287115 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:08.287792 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:08.287807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:08.289487 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:08.291245 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:08.291249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:08.291260 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:08.291508 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:08.291756 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:08.291758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:08.291759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:08.291899 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:08.292255 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:08.292259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:08.292487 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 12:25:08.292639 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 12:25:08.292884 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260620 12:25:08.294879 3297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42885
I20260620 12:25:08.295118 3297 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:08.295585 3297 heartbeater.cc:511] Master 127.0.98.62:42885 requested a full tablet report, sending...
I20260620 12:25:08.296573 2838 ts_manager.cc:194] Registered new tserver with Master: bf3e5d19586743d384872783f64ca83f (127.0.98.3:38393)
I20260620 12:25:08.297209 2838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46701
I20260620 12:25:08.299885 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:25:08.311957 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:08.310709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.0.98.62:42885)
0620 12:25:08.310836 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:08.310839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:08.310990 (+ 151us) server_negotiation.cc:407] Connection header received
0620 12:25:08.311081 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:08.311083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:08.311121 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:08.311193 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:08.311827 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55106: BlockingRecv error: recv got EOF from 127.0.0.1:55106 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:25:08.312960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 2900
I20260620 12:25:08.320209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3034
I20260620 12:25:08.326758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3168
I20260620 12:25:08.332157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 2806
2026-06-20T12:25:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2132 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:08 dist-test-slave-bwpp krb5kdc[3309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:08 dist-test-slave-bwpp krb5kdc[3309](info): set up 2 sockets
Jun 20 12:25:08 dist-test-slave-bwpp krb5kdc[3309](info): commencing operation
krb5kdc: starting...
W20260620 12:25:09.386544 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:09 dist-test-slave-bwpp krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:09.502846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42755
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:09.608960 3322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:09.609222 3322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:09.609262 3322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:09.613168 3322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:09.613217 3322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:09.613236 3322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:09.613253 3322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:09.613268 3322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:09.618360 3322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:09.619331 3322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:09.620296 3322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:09.628611 3330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:09.628592 3327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:09.628650 3322 server_base.cc:1034] running on GCE node
W20260620 12:25:09.628594 3328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:09.629366 3322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:09.630404 3322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:09.631590 3322 hybrid_clock.cc:648] HybridClock initialized: now 1781958309631562 us; error 40 us; skew 500 ppm
Jun 20 12:25:09 dist-test-slave-bwpp krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:09.634111 3322 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:09.635035 3322 webserver.cc:460] Webserver started at http://127.0.98.62:40613/ using document root <none> and password file <none>
I20260620 12:25:09.635577 3322 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:09.635665 3322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:09.635888 3322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:09.638151 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "04952def4ee04c6fa9c7130847bdf11f"
format_stamp: "Formatted at 2026-06-20 12:25:09 on dist-test-slave-bwpp"
server_key: "29100296982cc28cb7a64dac1ba77f65"
server_key_iv: "b3e91b192514917be27c39d3a4118bde"
server_key_version: "encryptionkey@0"
I20260620 12:25:09.638779 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "04952def4ee04c6fa9c7130847bdf11f"
format_stamp: "Formatted at 2026-06-20 12:25:09 on dist-test-slave-bwpp"
server_key: "29100296982cc28cb7a64dac1ba77f65"
server_key_iv: "b3e91b192514917be27c39d3a4118bde"
server_key_version: "encryptionkey@0"
I20260620 12:25:09.642808 3322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:25:09.645156 3337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:09.646358 3322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:09.646499 3322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "04952def4ee04c6fa9c7130847bdf11f"
format_stamp: "Formatted at 2026-06-20 12:25:09 on dist-test-slave-bwpp"
server_key: "29100296982cc28cb7a64dac1ba77f65"
server_key_iv: "b3e91b192514917be27c39d3a4118bde"
server_key_version: "encryptionkey@0"
I20260620 12:25:09.646607 3322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:09.660768 3322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:09.663421 3322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:09.663617 3322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:09.670397 3322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42755
I20260620 12:25:09.670696 3389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42755 every 8 connection(s)
I20260620 12:25:09.671392 3322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:09.673938 3390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:09.678438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3322
I20260620 12:25:09.678555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:09.678858 392 external_mini_cluster.cc:1428] Setting key 033a28bcb206e8a69d8c6786318d554f
I20260620 12:25:09.679327 3390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f: Bootstrap starting.
I20260620 12:25:09.681584 3390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:09.682494 3390 log.cc:824] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:09.684428 3390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f: No bootstrap required, opened a new log
Jun 20 12:25:09 dist-test-slave-bwpp krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:09.688661 3390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04952def4ee04c6fa9c7130847bdf11f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42755 } }
I20260620 12:25:09.688916 3390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:09.689011 3390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04952def4ee04c6fa9c7130847bdf11f, State: Initialized, Role: FOLLOWER
I20260620 12:25:09.689539 3390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04952def4ee04c6fa9c7130847bdf11f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42755 } }
I20260620 12:25:09.689703 3390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:09.689766 3390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:09.689865 3390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:09.690941 3390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04952def4ee04c6fa9c7130847bdf11f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42755 } }
I20260620 12:25:09.691282 3390 leader_election.cc:302] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04952def4ee04c6fa9c7130847bdf11f; no voters:
I20260620 12:25:09.691614 3390 leader_election.cc:288] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:09.691677 3395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:09.691910 3395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [term 1 LEADER]: Becoming Leader. State: Replica: 04952def4ee04c6fa9c7130847bdf11f, State: Running, Role: LEADER
I20260620 12:25:09.692252 3395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04952def4ee04c6fa9c7130847bdf11f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42755 } }
I20260620 12:25:09.692864 3390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:09.693434 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04952def4ee04c6fa9c7130847bdf11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04952def4ee04c6fa9c7130847bdf11f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42755 } } }
I20260620 12:25:09.693588 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:09.680070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.0.98.62:42755)
0620 12:25:09.680458 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:09.680473 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:09.680513 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:25:09.681105 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:09.681121 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:09.681452 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:09.681792 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:09.682725 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:09.683837 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:09.684949 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:09.685395 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:09.685519 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:09.688148 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:09.688153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:09.688165 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:09.688190 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:09.690248 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:09.690697 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:09.690702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:09.690708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:09.690855 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:09.691149 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:09.691153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:09.691155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:09.691593 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:09.691808 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:09.692462 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 12:25:09.692693 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":128,"threads_started":1}
I20260620 12:25:09.693598 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:09.694062 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04952def4ee04c6fa9c7130847bdf11f. Latest consensus state: current_term: 1 leader_uuid: "04952def4ee04c6fa9c7130847bdf11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04952def4ee04c6fa9c7130847bdf11f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42755 } } }
I20260620 12:25:09.694173 3404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:09.694234 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:09.696519 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:25:09.697378 3404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:09.705006 3404 catalog_manager.cc:1347] Generated new cluster ID: ee373e18c8cf457b843aa195967af5ba
I20260620 12:25:09.705067 3404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:09.712970 3404 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:09.713830 3404 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:09.719344 3404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04952def4ee04c6fa9c7130847bdf11f: Generated new TSK 0
I20260620 12:25:09.720149 3404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:09.721549 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:09.716165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.0.98.62:42755)
0620 12:25:09.716260 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:09.716264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:09.716516 (+ 252us) server_negotiation.cc:407] Connection header received
0620 12:25:09.716638 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:09.716641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:09.716684 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:09.716771 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:09.717328 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:09.718057 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:09.719158 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:09.719526 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:09.719627 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:09.720560 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:09.720565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:09.720568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:09.721089 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:09.721123 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:09.721143 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:09.721232 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:09.721349 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 12:25:09.721409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":33}
I20260620 12:25:09.726213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3322
2026-06-20T12:25:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1391 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:09 dist-test-slave-bwpp krb5kdc[3423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:09 dist-test-slave-bwpp krb5kdc[3423](info): set up 2 sockets
Jun 20 12:25:09 dist-test-slave-bwpp krb5kdc[3423](info): commencing operation
krb5kdc: starting...
W20260620 12:25:10.778607 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:10 dist-test-slave-bwpp krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:10.898232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:11.005517 3436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:11.005789 3436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:11.005831 3436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:11.009835 3436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:11.009896 3436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:11.009922 3436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:11.009939 3436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:11.009960 3436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:11.015249 3436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:11.016248 3436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:11.017191 3436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:11.025566 3441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:11.025599 3442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:11.025696 3436 server_base.cc:1034] running on GCE node
W20260620 12:25:11.025573 3444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:11.026299 3436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:11.027429 3436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:11.028586 3436 hybrid_clock.cc:648] HybridClock initialized: now 1781958311028555 us; error 47 us; skew 500 ppm
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:11.031258 3436 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:11.032130 3436 webserver.cc:460] Webserver started at http://127.0.98.62:43057/ using document root <none> and password file <none>
I20260620 12:25:11.032675 3436 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:11.032756 3436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:11.033010 3436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:11.035063 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "0fdf6d6398a94568baa5442a1d4a4c00"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "7e845c22bab96517dd017084285ce112"
server_key_iv: "e61d2535783483ca0b2ad92e6d6dc2b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.035602 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "0fdf6d6398a94568baa5442a1d4a4c00"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "7e845c22bab96517dd017084285ce112"
server_key_iv: "e61d2535783483ca0b2ad92e6d6dc2b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.039100 3436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:11.041338 3451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.042519 3436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:11.042680 3436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "0fdf6d6398a94568baa5442a1d4a4c00"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "7e845c22bab96517dd017084285ce112"
server_key_iv: "e61d2535783483ca0b2ad92e6d6dc2b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.042788 3436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:11.052940 3436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:11.055884 3436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:11.056044 3436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:11.062878 3436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44049
I20260620 12:25:11.063174 3503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44049 every 8 connection(s)
I20260620 12:25:11.063832 3436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:11.066476 3504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:11.071904 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00: Bootstrap starting.
I20260620 12:25:11.073657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3436
I20260620 12:25:11.073747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:11.073988 392 external_mini_cluster.cc:1428] Setting key 54ae760890934f3df72b5aae0276cb38
I20260620 12:25:11.074502 3504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:11.075363 3504 log.cc:824] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:11.077471 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00: No bootstrap required, opened a new log
I20260620 12:25:11.081884 3504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260620 12:25:11.082118 3504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:11.082191 3504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdf6d6398a94568baa5442a1d4a4c00, State: Initialized, Role: FOLLOWER
I20260620 12:25:11.082645 3504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260620 12:25:11.082863 3504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:11.082963 3504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:11.083099 3504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:11.084162 3504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260620 12:25:11.084553 3504 leader_election.cc:302] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fdf6d6398a94568baa5442a1d4a4c00; no voters:
I20260620 12:25:11.084915 3504 leader_election.cc:288] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:11.085148 3509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:11.085402 3509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [term 1 LEADER]: Becoming Leader. State: Replica: 0fdf6d6398a94568baa5442a1d4a4c00, State: Running, Role: LEADER
I20260620 12:25:11.085722 3509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260620 12:25:11.086345 3504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:11.086927 3511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fdf6d6398a94568baa5442a1d4a4c00. Latest consensus state: current_term: 1 leader_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } } }
I20260620 12:25:11.087637 3511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:11.087536 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdf6d6398a94568baa5442a1d4a4c00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } } }
I20260620 12:25:11.087963 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:11.088666 3518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:11.089551 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.075143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.0.98.62:44049)
0620 12:25:11.075624 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:11.075643 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:11.075690 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:25:11.076484 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:11.076521 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:11.076913 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:11.077266 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:11.077964 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.079516 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.080335 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.081110 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.081266 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.084019 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:11.084028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:11.084050 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:11.084087 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:11.086096 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.086596 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.086602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.086608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.086774 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.087096 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.087101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.087102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.087677 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:11.087877 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:11.088327 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 12:25:11.088626 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":138,"threads_started":1}
I20260620 12:25:11.091660 3518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:11.096575 3518 catalog_manager.cc:1347] Generated new cluster ID: 7e312c7e855d4f109c1f1f9d165625b1
I20260620 12:25:11.096638 3518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:11.130234 3518 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:11.131609 3518 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:11.139003 3518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fdf6d6398a94568baa5442a1d4a4c00: Generated new TSK 0
I20260620 12:25:11.139681 3518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:25:11.171329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:11.310108 3530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:11.310411 3530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:11.310487 3530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:11.314587 3530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:11.314698 3530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:11.314800 3530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:11.320346 3530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:11.321465 3530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:11.322510 3530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:11.332159 3535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:11.332175 3538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:11.332231 3530 server_base.cc:1034] running on GCE node
W20260620 12:25:11.332180 3536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:11.332883 3530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:11.333458 3530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:11.334638 3530 hybrid_clock.cc:648] HybridClock initialized: now 1781958311334613 us; error 53 us; skew 500 ppm
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:11.343789 3530 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:11.344732 3530 webserver.cc:460] Webserver started at http://127.0.98.1:40875/ using document root <none> and password file <none>
I20260620 12:25:11.345320 3530 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:11.345417 3530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:11.345702 3530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:11.347972 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "76a2f906cddb443f9f8b82a27750022c"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "cb1dd8571366cfcfc523207d58198ab2"
server_key_iv: "8805f8c9e3646a1ae0ba57ad3bc9eb64"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.348496 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "76a2f906cddb443f9f8b82a27750022c"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "cb1dd8571366cfcfc523207d58198ab2"
server_key_iv: "8805f8c9e3646a1ae0ba57ad3bc9eb64"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.352286 3530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260620 12:25:11.354697 3545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.355962 3530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 12:25:11.356092 3530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "76a2f906cddb443f9f8b82a27750022c"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "cb1dd8571366cfcfc523207d58198ab2"
server_key_iv: "8805f8c9e3646a1ae0ba57ad3bc9eb64"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.356202 3530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:11.368386 3530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:11.371645 3530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:11.371850 3530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:11.372428 3530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:11.373255 3530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:11.373328 3530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.373399 3530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:11.373456 3530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.382433 3530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36723
I20260620 12:25:11.382473 3658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36723 every 8 connection(s)
I20260620 12:25:11.383323 3530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:11.388073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3530
I20260620 12:25:11.388226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:11.388540 392 external_mini_cluster.cc:1428] Setting key e137f27d394ce5e5ef090a577233a098
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:11.398166 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.385062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50339 (local address 127.0.98.62:44049)
0620 12:25:11.385207 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:11.385212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:11.386127 (+ 915us) server_negotiation.cc:407] Connection header received
0620 12:25:11.387643 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:11.387645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:11.387745 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:11.387821 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:11.389351 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.389965 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.391194 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.391551 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.391665 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.394235 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:11.394242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:11.394248 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:11.394280 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:11.396085 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.396566 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.396570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.396572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.396670 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.396992 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.396997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.397000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.397240 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:11.397354 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:11.397817 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 12:25:11.397975 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:25:11.398974 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.385521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44049 (local address 127.0.98.1:50339)
0620 12:25:11.385966 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:11.386017 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:11.387405 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:11.388060 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:11.388070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:11.388604 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:11.389191 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:11.389205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.390097 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.390101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:11.391067 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:11.391076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.391723 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.391727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:11.391907 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.392558 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:11.392574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:11.394059 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:11.396200 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:11.396204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:11.396217 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:11.396457 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:11.396769 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:11.396771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:11.396773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:11.396897 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:11.397341 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:11.397346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:11.397572 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 12:25:11.397733 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 12:25:11.397985 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260620 12:25:11.400337 3659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44049
I20260620 12:25:11.400597 3659 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:11.401188 3659 heartbeater.cc:511] Master 127.0.98.62:44049 requested a full tablet report, sending...
I20260620 12:25:11.402882 3468 ts_manager.cc:194] Registered new tserver with Master: 76a2f906cddb443f9f8b82a27750022c (127.0.98.1:36723)
I20260620 12:25:11.404011 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:11.415347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:11.524986 3664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:11.525234 3664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:11.525285 3664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:11.529353 3664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:11.529409 3664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:11.529512 3664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:11.534711 3664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:11.535806 3664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:11.536814 3664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:11.546206 3672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:11.546198 3669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:11.546248 3670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:11.546751 3664 server_base.cc:1034] running on GCE node
I20260620 12:25:11.547169 3664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:11.547734 3664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:11.548887 3664 hybrid_clock.cc:648] HybridClock initialized: now 1781958311548874 us; error 33 us; skew 500 ppm
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:11.551517 3664 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:11.552610 3664 webserver.cc:460] Webserver started at http://127.0.98.2:35275/ using document root <none> and password file <none>
I20260620 12:25:11.553113 3664 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:11.553170 3664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:11.553323 3664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:11.555418 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "a941b23c039d441392ce05b6b1d5ce3a"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "2ffba8cfa29ceabd59f9ee4b2635ffd4"
server_key_iv: "f4029e460f6018f223da8e4df0cceeb2"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.555910 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a941b23c039d441392ce05b6b1d5ce3a"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "2ffba8cfa29ceabd59f9ee4b2635ffd4"
server_key_iv: "f4029e460f6018f223da8e4df0cceeb2"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.559509 3664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:11.561864 3679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.563113 3664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:11.563231 3664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "a941b23c039d441392ce05b6b1d5ce3a"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "2ffba8cfa29ceabd59f9ee4b2635ffd4"
server_key_iv: "f4029e460f6018f223da8e4df0cceeb2"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.563329 3664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:11.604066 3664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:11.607151 3664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:11.607347 3664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:11.607954 3664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:11.608762 3664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:11.608806 3664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.608868 3664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:11.608902 3664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.617935 3664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36317
I20260620 12:25:11.617955 3792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36317 every 8 connection(s)
I20260620 12:25:11.618891 3664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:11.621977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3664
I20260620 12:25:11.622066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:11.622334 392 external_mini_cluster.cc:1428] Setting key 05d182e588b6c09773d3c4610c1fd5fe
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:25:11.633265 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.620661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34555 (local address 127.0.98.62:44049)
0620 12:25:11.620807 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:11.620811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:11.621607 (+ 796us) server_negotiation.cc:407] Connection header received
0620 12:25:11.623228 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:11.623231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:11.623282 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:11.623397 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:11.624572 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.625270 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.626417 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.626869 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.626991 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.629800 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:11.629803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:11.629806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:11.629828 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:11.631306 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.631918 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.631920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.631922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.632002 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.632300 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.632302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.632303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.632462 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:11.632536 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:11.633047 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 12:25:11.633143 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:25:11.634303 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.621037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44049 (local address 127.0.98.2:34555)
0620 12:25:11.621466 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:11.621505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:11.623027 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:11.623570 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:11.623576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:11.623971 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:11.624435 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:11.624447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.625406 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.625408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:11.626291 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:11.626300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.627014 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.627017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:11.627157 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.627813 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:11.627829 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:11.629641 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:11.631421 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:11.631426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:11.631441 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:11.631815 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:11.632089 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:11.632091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:11.632092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:11.632221 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:11.632603 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:11.632608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:11.632827 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 12:25:11.632984 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 12:25:11.633218 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260620 12:25:11.636379 3793 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44049
I20260620 12:25:11.636622 3793 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:11.637182 3793 heartbeater.cc:511] Master 127.0.98.62:44049 requested a full tablet report, sending...
I20260620 12:25:11.638324 3468 ts_manager.cc:194] Registered new tserver with Master: a941b23c039d441392ce05b6b1d5ce3a (127.0.98.2:36317)
I20260620 12:25:11.639058 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:11.649612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:11.755702 3798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:11.755937 3798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:11.755977 3798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:11.760133 3798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:11.760192 3798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:11.760248 3798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:11.765406 3798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:11.766425 3798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:11.767496 3798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:11.776772 3803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:11.776955 3798 server_base.cc:1034] running on GCE node
W20260620 12:25:11.776772 3806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:11.776768 3804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:11.777503 3798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:11.778120 3798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:11.779301 3798 hybrid_clock.cc:648] HybridClock initialized: now 1781958311779289 us; error 26 us; skew 500 ppm
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:11.781946 3798 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:11.782845 3798 webserver.cc:460] Webserver started at http://127.0.98.3:34531/ using document root <none> and password file <none>
I20260620 12:25:11.783390 3798 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:11.783479 3798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:11.783703 3798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:11.785738 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "d81b07475ee74cd9a92318cf64a36484"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "e4a9847b900b2028cdb38c45f938562a"
server_key_iv: "3f8a47879abef30fd4e4026f22947c9a"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.786305 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d81b07475ee74cd9a92318cf64a36484"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "e4a9847b900b2028cdb38c45f938562a"
server_key_iv: "3f8a47879abef30fd4e4026f22947c9a"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.790203 3798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:25:11.792629 3813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.794157 3798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:25:11.794286 3798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "d81b07475ee74cd9a92318cf64a36484"
format_stamp: "Formatted at 2026-06-20 12:25:11 on dist-test-slave-bwpp"
server_key: "e4a9847b900b2028cdb38c45f938562a"
server_key_iv: "3f8a47879abef30fd4e4026f22947c9a"
server_key_version: "encryptionkey@0"
I20260620 12:25:11.794404 3798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:11.816653 3798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:11.820004 3798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:11.820152 3798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:11.820748 3798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:11.821543 3798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:11.821589 3798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.821655 3798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:11.821692 3798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:11.830529 3798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46293
I20260620 12:25:11.830612 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46293 every 8 connection(s)
I20260620 12:25:11.831475 3798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:11.835566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3798
I20260620 12:25:11.835702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:11.836032 392 external_mini_cluster.cc:1428] Setting key ce83ae51ba210a02e799a66fd3127c00
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:11.844730 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.833287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43629 (local address 127.0.98.62:44049)
0620 12:25:11.833438 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:11.833442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:11.834328 (+ 886us) server_negotiation.cc:407] Connection header received
0620 12:25:11.835826 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:11.835831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:11.835893 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:11.836002 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:11.837129 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.837659 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.838833 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.839059 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.839165 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.841401 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:11.841404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:11.841406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:11.841429 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:11.842735 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.843227 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.843232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.843234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.843314 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.843709 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.843714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.843716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.843889 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:11.844017 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:11.844467 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 12:25:11.844577 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:25:11.845526 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.833717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44049 (local address 127.0.98.3:43629)
0620 12:25:11.834181 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:11.834221 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:11.835597 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:11.836177 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:11.836183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:11.836522 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:11.836989 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:11.837001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.837797 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.837800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:11.838707 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:11.838721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.839208 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.839211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:11.839379 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.839882 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:11.839898 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:11.841212 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:11.842850 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:11.842855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:11.842868 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:11.843119 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:11.843413 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:11.843416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:11.843418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:11.843610 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:11.844014 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:11.844020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:11.844244 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 12:25:11.844429 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 12:25:11.844738 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260620 12:25:11.846547 3927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44049
I20260620 12:25:11.846822 3927 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:11.847388 3927 heartbeater.cc:511] Master 127.0.98.62:44049 requested a full tablet report, sending...
I20260620 12:25:11.848489 3468 ts_manager.cc:194] Registered new tserver with Master: d81b07475ee74cd9a92318cf64a36484 (127.0.98.3:46293)
I20260620 12:25:11.849273 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43629
I20260620 12:25:11.849608 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958311, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:11.873818 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:11.866166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.0.98.62:44049)
0620 12:25:11.866289 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:11.866292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:11.866470 (+ 178us) server_negotiation.cc:407] Connection header received
0620 12:25:11.866566 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:11.866569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:11.866606 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:11.866748 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:11.867451 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.867949 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.868712 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:11.868909 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:11.868988 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:11.871259 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:11.871262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:11.871264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:11.871284 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:11.872484 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.872848 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.872850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.872852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.872919 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:11.873133 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:11.873135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:11.873136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:11.873324 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:11.873424 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:11.873613 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 12:25:11.873701 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 12:25:11.874204 3468 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59124
I20260620 12:25:11.875021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3530
I20260620 12:25:11.880657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3664
I20260620 12:25:11.885846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3798
I20260620 12:25:11.891134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3436
2026-06-20T12:25:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2168 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3939](info): set up 2 sockets
Jun 20 12:25:11 dist-test-slave-bwpp krb5kdc[3939](info): commencing operation
krb5kdc: starting...
W20260620 12:25:12.926752 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:13.043907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46445
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:13.150835 3952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:13.151090 3952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:13.151175 3952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:13.155557 3952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:13.155620 3952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:13.155642 3952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:13.155695 3952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:13.155730 3952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:13.160981 3952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:13.161931 3952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:13.162959 3952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:13.171020 3960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:13.171100 3952 server_base.cc:1034] running on GCE node
W20260620 12:25:13.171020 3957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:13.171038 3958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:13.171907 3952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:13.172912 3952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:13.174098 3952 hybrid_clock.cc:648] HybridClock initialized: now 1781958313174087 us; error 24 us; skew 500 ppm
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:13.176782 3952 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:13.177592 3952 webserver.cc:460] Webserver started at http://127.0.98.62:44895/ using document root <none> and password file <none>
I20260620 12:25:13.178119 3952 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:13.178206 3952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:13.178437 3952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:13.180399 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "2b228345949d4232b16a5e5dea30469d"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "a63fec76537c8f3d8101cb31514fd4d0"
server_key_iv: "326630cf7b7ef69a4f6a37badb5b9ffe"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.180960 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b228345949d4232b16a5e5dea30469d"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "a63fec76537c8f3d8101cb31514fd4d0"
server_key_iv: "326630cf7b7ef69a4f6a37badb5b9ffe"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.184511 3952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:13.186837 3967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.187984 3952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:13.188122 3952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "2b228345949d4232b16a5e5dea30469d"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "a63fec76537c8f3d8101cb31514fd4d0"
server_key_iv: "326630cf7b7ef69a4f6a37badb5b9ffe"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.188228 3952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:13.211127 3952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:13.214069 3952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:13.214270 3952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:13.221225 3952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46445
I20260620 12:25:13.221283 4019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46445 every 8 connection(s)
I20260620 12:25:13.222168 3952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:13.224936 4020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:13.229337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 3952
I20260620 12:25:13.229485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:13.229825 392 external_mini_cluster.cc:1428] Setting key 8c15c65c7956a517ab2be11b7b65fefa
I20260620 12:25:13.230858 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d: Bootstrap starting.
I20260620 12:25:13.233135 4020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:13.234050 4020 log.cc:824] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:13.236243 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d: No bootstrap required, opened a new log
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:13.240355 4020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b228345949d4232b16a5e5dea30469d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 12:25:13.240581 4020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:13.240660 4020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b228345949d4232b16a5e5dea30469d, State: Initialized, Role: FOLLOWER
I20260620 12:25:13.241076 4020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b228345949d4232b16a5e5dea30469d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 12:25:13.241216 4020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:13.241307 4020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:13.241436 4020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:13.242259 4020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b228345949d4232b16a5e5dea30469d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 12:25:13.242586 4020 leader_election.cc:302] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b228345949d4232b16a5e5dea30469d; no voters:
I20260620 12:25:13.242980 4020 leader_election.cc:288] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:13.243041 4025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:13.243288 4025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [term 1 LEADER]: Becoming Leader. State: Replica: 2b228345949d4232b16a5e5dea30469d, State: Running, Role: LEADER
I20260620 12:25:13.243623 4025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b228345949d4232b16a5e5dea30469d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 12:25:13.244266 4020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:13.244767 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:13.231249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.0.98.62:46445)
0620 12:25:13.231646 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:13.231661 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:13.231697 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:25:13.232296 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:13.232312 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:13.232632 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:13.232895 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:13.233871 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.235073 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.236191 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.236694 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.236810 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:13.239528 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:13.239533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:13.239545 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:13.239570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:13.241506 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.241932 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.241938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.241944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.242065 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.242348 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.242351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.242353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.242863 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:13.243063 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:13.243669 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 12:25:13.243943 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260620 12:25:13.245440 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b228345949d4232b16a5e5dea30469d. Latest consensus state: current_term: 1 leader_uuid: "2b228345949d4232b16a5e5dea30469d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b228345949d4232b16a5e5dea30469d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } } }
I20260620 12:25:13.245795 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:13.245708 4027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b228345949d4232b16a5e5dea30469d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b228345949d4232b16a5e5dea30469d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } } }
I20260620 12:25:13.246107 4027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d [sys.catalog]: This master's current role is: LEADER
W20260620 12:25:13.248687 4040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:25:13.248777 4040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:25:13.248853 4038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:13.249550 4038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:13.254947 4038 catalog_manager.cc:1347] Generated new cluster ID: 2b363a21a5d4419883bf353db9ae79b5
I20260620 12:25:13.255010 4038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:13.288879 4038 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:13.289837 4038 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:13.295018 4038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b228345949d4232b16a5e5dea30469d: Generated new TSK 0
I20260620 12:25:13.295694 4038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:25:13.326804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:13.438786 4046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:13.439039 4046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:13.439083 4046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:13.443084 4046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:13.443135 4046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:13.443190 4046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:13.448500 4046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:13.449432 4046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:13.450387 4046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:13.459918 4054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:13.459914 4051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:13.459911 4052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:13.460140 4046 server_base.cc:1034] running on GCE node
I20260620 12:25:13.460605 4046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:13.461282 4046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:13.462461 4046 hybrid_clock.cc:648] HybridClock initialized: now 1781958313462436 us; error 46 us; skew 500 ppm
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:13.465163 4046 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:13.466168 4046 webserver.cc:460] Webserver started at http://127.0.98.1:37805/ using document root <none> and password file <none>
I20260620 12:25:13.466771 4046 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:13.466836 4046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:13.467010 4046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:13.468940 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "d10b34701983442b913f44713fab71f6"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "87f796efb3f131777bd4eae094bb6678"
server_key_iv: "85f36698f50a98fd6cf8b5fd3f9002f6"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.469429 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d10b34701983442b913f44713fab71f6"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "87f796efb3f131777bd4eae094bb6678"
server_key_iv: "85f36698f50a98fd6cf8b5fd3f9002f6"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.473022 4046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 12:25:13.475281 4061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.476485 4046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:13.476627 4046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "d10b34701983442b913f44713fab71f6"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "87f796efb3f131777bd4eae094bb6678"
server_key_iv: "85f36698f50a98fd6cf8b5fd3f9002f6"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.476737 4046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:13.508466 4046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:13.511503 4046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:13.511708 4046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:13.512319 4046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:13.513131 4046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:13.513200 4046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.513271 4046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:13.513321 4046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.522037 4046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46007
I20260620 12:25:13.522061 4174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46007 every 8 connection(s)
I20260620 12:25:13.522984 4046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:13.525660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 4046
I20260620 12:25:13.525774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:13.526132 392 external_mini_cluster.cc:1428] Setting key adddbcc599db1b5d51fec0cabe914c52
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:13.539021 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:13.524734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49857 (local address 127.0.98.62:46445)
0620 12:25:13.524862 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:13.524867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:13.526019 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 12:25:13.527840 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:13.527846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:13.527910 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:13.528002 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:13.529702 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.530266 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.531517 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.531760 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.531882 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:13.535048 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:13.535051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:13.535054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:13.535075 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:13.536698 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.537231 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.537234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.537236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.537324 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.537733 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.537736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.537738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.537927 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:13.538033 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:13.538628 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 12:25:13.538841 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:25:13.539835 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:13.525258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.1:49857)
0620 12:25:13.525849 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:13.525908 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:13.527570 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:13.528210 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:13.528219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:13.528771 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:13.529504 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:13.529528 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.530408 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.530411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:13.531388 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:13.531398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.531913 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.531917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:13.532095 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:13.532773 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:13.532790 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:13.534849 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:13.536827 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:13.536833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:13.536849 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:13.537111 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:13.537425 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:13.537431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:13.537436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:13.537617 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:13.538043 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:13.538051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:13.538381 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 12:25:13.538574 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 12:25:13.538951 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":176,"threads_started":1}
I20260620 12:25:13.541252 4175 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46445
I20260620 12:25:13.541539 4175 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:13.542150 4175 heartbeater.cc:511] Master 127.0.98.62:46445 requested a full tablet report, sending...
I20260620 12:25:13.544256 3984 ts_manager.cc:194] Registered new tserver with Master: d10b34701983442b913f44713fab71f6 (127.0.98.1:46007)
I20260620 12:25:13.545912 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:13.555285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:13.663674 4180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:13.663925 4180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:13.663967 4180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:13.667959 4180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:13.668015 4180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:13.668073 4180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:13.673126 4180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:13.674079 4180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:13.675061 4180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:13.684551 4185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:13.684528 4186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:13.684525 4188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:13.684880 4180 server_base.cc:1034] running on GCE node
I20260620 12:25:13.685225 4180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:13.685775 4180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:13.686935 4180 hybrid_clock.cc:648] HybridClock initialized: now 1781958313686908 us; error 40 us; skew 500 ppm
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:13.689569 4180 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:13.690389 4180 webserver.cc:460] Webserver started at http://127.0.98.2:43017/ using document root <none> and password file <none>
I20260620 12:25:13.690970 4180 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:13.691031 4180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:13.691289 4180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:13.693315 4180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7e473b3f798411784d2ac3900eeecef"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "dfd3cdb0b40074f63e30d6b5ad9eec68"
server_key_iv: "bcec5002e400fab77e5d82952d3ec84f"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.693866 4180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7e473b3f798411784d2ac3900eeecef"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "dfd3cdb0b40074f63e30d6b5ad9eec68"
server_key_iv: "bcec5002e400fab77e5d82952d3ec84f"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.697414 4180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 12:25:13.699867 4195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.701097 4180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:13.701187 4180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "a7e473b3f798411784d2ac3900eeecef"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "dfd3cdb0b40074f63e30d6b5ad9eec68"
server_key_iv: "bcec5002e400fab77e5d82952d3ec84f"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.701316 4180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:13.731750 4180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:13.735394 4180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:13.735600 4180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:13.736225 4180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:13.737054 4180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:13.737100 4180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.737162 4180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:13.737200 4180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.746925 4180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40261
I20260620 12:25:13.746944 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40261 every 8 connection(s)
I20260620 12:25:13.747793 4180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:13.751528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 4180
I20260620 12:25:13.751613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:13.751900 392 external_mini_cluster.cc:1428] Setting key f5f9e79a9e2a5edc141afc9f87b4c642
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:25:13.762844 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:13.749499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52019 (local address 127.0.98.62:46445)
0620 12:25:13.749668 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:13.749673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:13.750586 (+ 913us) server_negotiation.cc:407] Connection header received
0620 12:25:13.752050 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:13.752053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:13.752111 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:13.752202 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:13.753481 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.754264 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.755442 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.755837 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.755965 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:13.758560 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:13.758563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:13.758565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:13.758588 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:13.760357 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.761023 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.761025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.761027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.761103 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.761540 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.761544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.761545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.761751 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:13.761875 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:13.762478 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 12:25:13.762583 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:25:13.763585 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:13.749915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.2:52019)
0620 12:25:13.750394 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:13.750435 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:13.751806 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:13.752406 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:13.752414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:13.752788 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:13.753342 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:13.753358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.754394 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.754396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:13.755316 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:13.755325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.755985 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.755988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:13.756123 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:13.756739 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:13.756755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:13.758379 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:13.760489 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:13.760496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:13.760513 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:13.760897 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:13.761209 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:13.761215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:13.761220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:13.761405 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:13.761892 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:13.761900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:13.762216 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 12:25:13.762391 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 12:25:13.762725 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:25:13.764819 4309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46445
I20260620 12:25:13.765055 4309 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:13.765539 4309 heartbeater.cc:511] Master 127.0.98.62:46445 requested a full tablet report, sending...
I20260620 12:25:13.766728 3984 ts_manager.cc:194] Registered new tserver with Master: a7e473b3f798411784d2ac3900eeecef (127.0.98.2:40261)
I20260620 12:25:13.767343 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:13.781891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:13.894438 4314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:13.894743 4314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:13.894791 4314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:13.898756 4314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:13.898823 4314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:13.898881 4314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:13.904691 4314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:13.905980 4314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:13.907069 4314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:13.916708 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:13.916747 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:13.916747 4320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:13.916990 4314 server_base.cc:1034] running on GCE node
I20260620 12:25:13.917508 4314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:13.918138 4314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:13.919317 4314 hybrid_clock.cc:648] HybridClock initialized: now 1781958313919302 us; error 29 us; skew 500 ppm
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:13.921984 4314 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:13.922931 4314 webserver.cc:460] Webserver started at http://127.0.98.3:35885/ using document root <none> and password file <none>
I20260620 12:25:13.923478 4314 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:13.923563 4314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:13.923782 4314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:13.925750 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "866471b3914749dc9ff75d1d30b49bd6"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "8c9ffaa71367f0fbb9b29e587f0cc003"
server_key_iv: "ba11ae38df68065d084a57864d6b78df"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.926281 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "866471b3914749dc9ff75d1d30b49bd6"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "8c9ffaa71367f0fbb9b29e587f0cc003"
server_key_iv: "ba11ae38df68065d084a57864d6b78df"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.929841 4314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:25:13.932193 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.933292 4314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:13.933429 4314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "866471b3914749dc9ff75d1d30b49bd6"
format_stamp: "Formatted at 2026-06-20 12:25:13 on dist-test-slave-bwpp"
server_key: "8c9ffaa71367f0fbb9b29e587f0cc003"
server_key_iv: "ba11ae38df68065d084a57864d6b78df"
server_key_version: "encryptionkey@0"
I20260620 12:25:13.933530 4314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:13.959141 4314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:13.962502 4314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:13.962769 4314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:13.963388 4314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:13.964255 4314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:13.964329 4314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.964396 4314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:13.964439 4314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:13.973646 4314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32873
I20260620 12:25:13.973670 4442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32873 every 8 connection(s)
I20260620 12:25:13.974545 4314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:13.978259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 4314
I20260620 12:25:13.978400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:13.978727 392 external_mini_cluster.cc:1428] Setting key a6b5d08d394ddad19398b4725526ea29
Jun 20 12:25:13 dist-test-slave-bwpp krb5kdc[3939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:13.988489 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:13.976390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50889 (local address 127.0.98.62:46445)
0620 12:25:13.976527 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:13.976530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:13.977347 (+ 817us) server_negotiation.cc:407] Connection header received
0620 12:25:13.978955 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:13.978958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:13.979012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:13.979120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:13.980367 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.980997 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.982139 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.982376 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.982494 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:13.984936 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:13.984940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:13.984943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:13.984968 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:13.986396 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.986992 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.986996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.986998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.987079 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:13.987420 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:13.987423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:13.987425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:13.987599 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:13.987706 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:13.988192 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 12:25:13.988306 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:25:13.989326 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:13.976781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.3:50889)
0620 12:25:13.977193 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:13.977233 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:13.978708 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:13.979298 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:13.979304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:13.979655 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:13.980213 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:13.980229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.981120 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.981126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:13.982012 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:13.982021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:13.982525 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:13.982529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:13.982717 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:13.983330 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:13.983346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:13.984771 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:13.986527 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:13.986533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:13.986547 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:13.986891 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:13.987173 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:13.987177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:13.987179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:13.987318 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:13.987703 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:13.987709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:13.987959 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 12:25:13.988138 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 12:25:13.988453 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260620 12:25:13.990453 4443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46445
I20260620 12:25:13.990741 4443 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:13.991336 4443 heartbeater.cc:511] Master 127.0.98.62:46445 requested a full tablet report, sending...
I20260620 12:25:13.992398 3984 ts_manager.cc:194] Registered new tserver with Master: 866471b3914749dc9ff75d1d30b49bd6 (127.0.98.3:32873)
I20260620 12:25:13.992944 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50889
I20260620 12:25:14.004057 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:25:14 dist-test-slave-bwpp krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:25:14 dist-test-slave-bwpp krb5kdc[3939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:25:14.029497 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:14.020175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.0.98.1:46007)
0620 12:25:14.020473 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:14.020487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:14.020511 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:25:14.020561 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:14.020565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:14.020755 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:14.020880 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:14.021522 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:14.022299 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:14.023147 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:14.023541 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:14.023623 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:14.025920 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:14.025929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:14.025941 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:14.025964 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:14.027687 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:14.028151 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:14.028156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:14.028157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:14.028232 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:14.028516 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:14.028519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:14.028520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:14.028757 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:14.029003 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:14.029160 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 12:25:14.029322 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
Jun 20 12:25:14 dist-test-slave-bwpp krb5kdc[3939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:14.040462 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:14.032247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.0.98.62:46445)
0620 12:25:14.032384 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:14.032389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:14.032492 (+ 103us) server_negotiation.cc:407] Connection header received
0620 12:25:14.032638 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:14.032641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:14.032690 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:14.032777 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:14.033353 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:14.033886 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:14.034695 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:14.034932 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:14.035044 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:14.037420 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:14.037423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:14.037425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:14.037448 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:14.039017 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:14.039369 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:14.039371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:14.039372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:14.039441 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:14.039655 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:14.039657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:14.039658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:14.039870 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:14.039972 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:14.040151 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:25:14.040276 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 12:25:14.040863 3984 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396
I20260620 12:25:14.041493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 4046
I20260620 12:25:14.046819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 4180
I20260620 12:25:14.051957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 4314
I20260620 12:25:14.057293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 3952
2026-06-20T12:25:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2165 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:14 dist-test-slave-bwpp krb5kdc[4455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:14 dist-test-slave-bwpp krb5kdc[4455](info): set up 2 sockets
Jun 20 12:25:14 dist-test-slave-bwpp krb5kdc[4455](info): commencing operation
krb5kdc: starting...
W20260620 12:25:15.111420 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:15 dist-test-slave-bwpp krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:15.227695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:15.335386 4468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:15.335672 4468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:15.335747 4468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:15.339725 4468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:15.339808 4468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:15.339849 4468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:15.339906 4468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:15.339944 4468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:15.344995 4468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:15.346020 4468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:15.347039 4468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:15.355225 4474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:15.355227 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:15.355217 4473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:15.355408 4468 server_base.cc:1034] running on GCE node
I20260620 12:25:15.356012 4468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:15.357113 4468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:15.358299 4468 hybrid_clock.cc:648] HybridClock initialized: now 1781958315358288 us; error 28 us; skew 500 ppm
Jun 20 12:25:15 dist-test-slave-bwpp krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:15.361014 4468 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:15.361753 4468 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:15.361845 4468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:15.362071 4468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:15.364193 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "c432108052f04771b9a54266a90061c8"
format_stamp: "Formatted at 2026-06-20 12:25:15 on dist-test-slave-bwpp"
server_key: "9a8bf944b7c4a83a1c8f2ded512e1c12"
server_key_iv: "68dfe37d161978feb7497ddae4cc6be5"
server_key_version: "encryptionkey@0"
I20260620 12:25:15.364748 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "c432108052f04771b9a54266a90061c8"
format_stamp: "Formatted at 2026-06-20 12:25:15 on dist-test-slave-bwpp"
server_key: "9a8bf944b7c4a83a1c8f2ded512e1c12"
server_key_iv: "68dfe37d161978feb7497ddae4cc6be5"
server_key_version: "encryptionkey@0"
I20260620 12:25:15.368391 4468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:15.370745 4482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:15.371867 4468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:15.372002 4468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "c432108052f04771b9a54266a90061c8"
format_stamp: "Formatted at 2026-06-20 12:25:15 on dist-test-slave-bwpp"
server_key: "9a8bf944b7c4a83a1c8f2ded512e1c12"
server_key_iv: "68dfe37d161978feb7497ddae4cc6be5"
server_key_version: "encryptionkey@0"
I20260620 12:25:15.372108 4468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:15.393566 4468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:15.396813 4468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:15.397023 4468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:15.404194 4468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36235
I20260620 12:25:15.404484 4534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36235 every 8 connection(s)
I20260620 12:25:15.404881 4468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:15.407626 4535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:15.413331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 4468
I20260620 12:25:15.413491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:15.413753 392 external_mini_cluster.cc:1428] Setting key b0a1d36e9dee821036a507c77b043638
I20260620 12:25:15.414542 4535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8: Bootstrap starting.
I20260620 12:25:15.417402 4535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:15.418349 4535 log.cc:824] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:15.420535 4535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8: No bootstrap required, opened a new log
Jun 20 12:25:15 dist-test-slave-bwpp krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:15.425753 4535 raft_consensus.cc:348] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c432108052f04771b9a54266a90061c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260620 12:25:15.426055 4535 raft_consensus.cc:374] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:15.426123 4535 raft_consensus.cc:729] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c432108052f04771b9a54266a90061c8, State: Initialized, Role: FOLLOWER
I20260620 12:25:15.426633 4535 consensus_queue.cc:260] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c432108052f04771b9a54266a90061c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260620 12:25:15.426883 4535 raft_consensus.cc:388] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:15.426961 4535 raft_consensus.cc:482] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:15.427078 4535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:15.429437 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:15.415208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.0.98.62:36235)
0620 12:25:15.415697 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:15.415721 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:15.415781 (+ 60us) server_negotiation.cc:407] Connection header received
0620 12:25:15.416646 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:15.416669 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:15.417090 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:15.417501 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:15.418167 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:15.419634 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:15.420496 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:15.421230 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:15.421401 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:15.423904 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:15.423913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:15.423937 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:15.423981 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:15.426274 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:15.426779 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:15.426785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:15.426791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:15.426956 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:15.427298 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:15.427304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:15.427309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:15.427865 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:15.428078 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:15.428302 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 12:25:15.428643 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":135,"threads_started":1}
I20260620 12:25:15.433030 4535 raft_consensus.cc:504] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c432108052f04771b9a54266a90061c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260620 12:25:15.433475 4535 leader_election.cc:302] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c432108052f04771b9a54266a90061c8; no voters:
I20260620 12:25:15.433838 4535 leader_election.cc:288] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:15.433950 4540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:15.434195 4540 raft_consensus.cc:686] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [term 1 LEADER]: Becoming Leader. State: Replica: c432108052f04771b9a54266a90061c8, State: Running, Role: LEADER
I20260620 12:25:15.434574 4540 consensus_queue.cc:237] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c432108052f04771b9a54266a90061c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260620 12:25:15.435245 4535 sys_catalog.cc:564] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:15.435824 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c432108052f04771b9a54266a90061c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c432108052f04771b9a54266a90061c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } } }
I20260620 12:25:15.435941 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:15.436622 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c432108052f04771b9a54266a90061c8. Latest consensus state: current_term: 1 leader_uuid: "c432108052f04771b9a54266a90061c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c432108052f04771b9a54266a90061c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } } }
I20260620 12:25:15.436703 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:15.437101 4544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:15.440835 4544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:15.445991 4544 catalog_manager.cc:1347] Generated new cluster ID: 9d8695d3235e4dd2a492643b24feacce
I20260620 12:25:15.446053 4544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:15.462431 4544 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:15.463457 4544 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:15.473278 4544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8: Generated new TSK 0
I20260620 12:25:15.473999 4544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260620 12:25:15.937716 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.459s user 0.003s sys 0.000s
I20260620 12:25:15.938221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:16.047398 4561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:16.047639 4561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:16.047683 4561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:16.051545 4561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:16.051594 4561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:16.051649 4561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:16.056752 4561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:16.057688 4561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:16.058699 4561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:16.067803 4566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:16.067830 4567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:16.068002 4561 server_base.cc:1034] running on GCE node
W20260620 12:25:16.068286 4569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:16.068739 4561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:16.069337 4561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:16.070487 4561 hybrid_clock.cc:648] HybridClock initialized: now 1781958316070468 us; error 30 us; skew 500 ppm
Jun 20 12:25:16 dist-test-slave-bwpp krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:16.073196 4561 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:16.073889 4561 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:16.073946 4561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:16.074107 4561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:16.076082 4561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "cf1795bbd0bc47aa961c2ff621b8a896"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "f201294e5faad4db95e1689ec50b99d9"
server_key_iv: "6186ef9a89f3f06e5e7e36fa06320890"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.076624 4561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cf1795bbd0bc47aa961c2ff621b8a896"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "f201294e5faad4db95e1689ec50b99d9"
server_key_iv: "6186ef9a89f3f06e5e7e36fa06320890"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.080144 4561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 12:25:16.082286 4575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.083500 4561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:25:16.083637 4561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "cf1795bbd0bc47aa961c2ff621b8a896"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "f201294e5faad4db95e1689ec50b99d9"
server_key_iv: "6186ef9a89f3f06e5e7e36fa06320890"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.083765 4561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:16.125299 4561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:16.128232 4561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:16.128432 4561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:16.129005 4561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:16.129830 4561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:16.129901 4561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.129971 4561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:16.130020 4561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.139050 4561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41371
I20260620 12:25:16.139067 4688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41371 every 8 connection(s)
I20260620 12:25:16.139613 4561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:16.144688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 4561
I20260620 12:25:16.144804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:16.145165 392 external_mini_cluster.cc:1428] Setting key d82b03647580fef1bfcb42b4ef21b3f3
Jun 20 12:25:16 dist-test-slave-bwpp krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:16.156582 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.141271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39431 (local address 127.0.98.62:36235)
0620 12:25:16.141509 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:16.141513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:16.142223 (+ 710us) server_negotiation.cc:407] Connection header received
0620 12:25:16.144318 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:16.144323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:16.144392 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:16.144504 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:16.146255 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.147060 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.148736 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.149166 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.149282 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.152641 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:16.152648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:16.152654 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:16.152686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:16.154430 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.154982 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.154985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.154987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.155085 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.155418 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.155422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.155424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.155675 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:16.155781 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:16.156266 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 12:25:16.156395 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:25:16.157208 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.141653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36235 (local address 127.0.98.1:39431)
0620 12:25:16.142065 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:16.142116 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:16.144047 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:16.144839 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:16.144852 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:16.145394 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:16.146103 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.146125 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.147202 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.147230 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.148599 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.148610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.149375 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.149384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.149648 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.150453 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:16.150474 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:16.152467 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:16.154542 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:16.154547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:16.154557 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:16.154883 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:16.155186 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:16.155188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:16.155190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:16.155312 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:16.155776 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:16.155780 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:16.156005 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 12:25:16.156150 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 12:25:16.156384 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260620 12:25:16.158380 4689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36235
I20260620 12:25:16.158727 4689 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:16.159289 4689 heartbeater.cc:511] Master 127.0.98.62:36235 requested a full tablet report, sending...
I20260620 12:25:16.161029 4496 ts_manager.cc:194] Registered new tserver with Master: cf1795bbd0bc47aa961c2ff621b8a896 (127.0.98.1:41371)
I20260620 12:25:16.162137 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:16.173790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:16.281518 4695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:16.281749 4695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:16.281793 4695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:16.286278 4695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:16.286345 4695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:16.286405 4695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:16.291587 4695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:16.292570 4695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:16.293560 4695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:16.303376 4701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:16.303378 4700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:16.303543 4695 server_base.cc:1034] running on GCE node
W20260620 12:25:16.303929 4703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:16.304358 4695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:16.305016 4695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:16.306217 4695 hybrid_clock.cc:648] HybridClock initialized: now 1781958316306204 us; error 28 us; skew 500 ppm
Jun 20 12:25:16 dist-test-slave-bwpp krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:16.309278 4695 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:16.310066 4695 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:16.310129 4695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:16.310341 4695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:16.312414 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad0bd64774b24b57a936269268a40029"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "74686b848426260299c850351fac28df"
server_key_iv: "039a1d6fb64ba297ea324c58696c1615"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.312981 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad0bd64774b24b57a936269268a40029"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "74686b848426260299c850351fac28df"
server_key_iv: "039a1d6fb64ba297ea324c58696c1615"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.316706 4695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:16.319198 4709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.320458 4695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:16.320592 4695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "ad0bd64774b24b57a936269268a40029"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "74686b848426260299c850351fac28df"
server_key_iv: "039a1d6fb64ba297ea324c58696c1615"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.320701 4695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:16.354807 4695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:16.357777 4695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:16.357977 4695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:16.358559 4695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:16.359488 4695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:16.359560 4695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.359627 4695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:16.359678 4695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.368279 4695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39473
I20260620 12:25:16.368295 4822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39473 every 8 connection(s)
I20260620 12:25:16.368861 4695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:16.370066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 4695
I20260620 12:25:16.370201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:16.370534 392 external_mini_cluster.cc:1428] Setting key 5e4241aeae0c0c28b3e27a1f358602f5
Jun 20 12:25:16 dist-test-slave-bwpp krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:25:16.384958 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.370677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56075 (local address 127.0.98.62:36235)
0620 12:25:16.370818 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:16.370822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:16.371678 (+ 856us) server_negotiation.cc:407] Connection header received
0620 12:25:16.373077 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:16.373083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:16.373144 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:16.373241 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:16.374911 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.375451 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.377036 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.377257 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.377340 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.380207 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:16.380216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:16.380218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:16.380262 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:16.381697 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.382311 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.382314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.382315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.382388 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.383853 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.383856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.383857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.384040 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:16.384152 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:16.384650 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 12:25:16.384776 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:25:16.385740 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.371066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36235 (local address 127.0.98.2:56075)
0620 12:25:16.371534 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:16.371585 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:16.372869 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:16.373469 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:16.373482 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:16.374027 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:16.374762 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.374779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.375585 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.375588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.376898 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.376914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.377422 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.377430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.377703 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.378327 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:16.378349 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:16.380045 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:16.381878 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:16.381884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:16.381899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:16.382159 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:16.382480 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:16.382483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:16.382485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:16.382622 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:16.384150 (+ 1528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:16.384157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:16.384412 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 12:25:16.384598 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 12:25:16.384917 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20260620 12:25:16.387123 4823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36235
I20260620 12:25:16.387400 4823 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:16.387976 4823 heartbeater.cc:511] Master 127.0.98.62:36235 requested a full tablet report, sending...
I20260620 12:25:16.389133 4496 ts_manager.cc:194] Registered new tserver with Master: ad0bd64774b24b57a936269268a40029 (127.0.98.2:39473)
I20260620 12:25:16.389983 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:16.397468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:16.507642 4828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:16.507884 4828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:16.507928 4828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:16.511986 4828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:16.512039 4828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:16.512144 4828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:16.517333 4828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:16.518329 4828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:16.519395 4828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:25:16.528540 4828 server_base.cc:1034] running on GCE node
W20260620 12:25:16.528626 4836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:16.528669 4833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:16.528626 4834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:16.529213 4828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:16.529795 4828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:16.530973 4828 hybrid_clock.cc:648] HybridClock initialized: now 1781958316530956 us; error 29 us; skew 500 ppm
Jun 20 12:25:16 dist-test-slave-bwpp krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:16.533566 4828 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:16.534276 4828 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:16.534370 4828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:16.534591 4828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:16.536666 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e11dfbc56a84463acc62587824dca7d"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "6d3bb19ec5c3c910cba70e8adf4db817"
server_key_iv: "7a2c1ff0f1621c9ca0c6c83dff9cb527"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.537293 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e11dfbc56a84463acc62587824dca7d"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "6d3bb19ec5c3c910cba70e8adf4db817"
server_key_iv: "7a2c1ff0f1621c9ca0c6c83dff9cb527"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.540736 4828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:16.543123 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.544364 4828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:16.544497 4828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "3e11dfbc56a84463acc62587824dca7d"
format_stamp: "Formatted at 2026-06-20 12:25:16 on dist-test-slave-bwpp"
server_key: "6d3bb19ec5c3c910cba70e8adf4db817"
server_key_iv: "7a2c1ff0f1621c9ca0c6c83dff9cb527"
server_key_version: "encryptionkey@0"
I20260620 12:25:16.544601 4828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:16.564591 4828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:16.567659 4828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:16.567857 4828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:16.568434 4828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:16.569260 4828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:16.569332 4828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.569398 4828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:16.569445 4828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:16.578962 4828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34609
I20260620 12:25:16.579021 4955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34609 every 8 connection(s)
I20260620 12:25:16.579550 4828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:16.583442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 4828
I20260620 12:25:16.583586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:16.583864 392 external_mini_cluster.cc:1428] Setting key 47119bb4efe9e33ae18d24a0f567923d
Jun 20 12:25:16 dist-test-slave-bwpp krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:16.593467 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.581194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54305 (local address 127.0.98.62:36235)
0620 12:25:16.581339 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:16.581343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:16.582232 (+ 889us) server_negotiation.cc:407] Connection header received
0620 12:25:16.583894 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:16.583898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:16.583962 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:16.584073 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:16.585517 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.586158 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.587495 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.587732 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.587858 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.590248 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:16.590251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:16.590253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:16.590275 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:16.591585 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.592057 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.592060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.592061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.592130 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.592581 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.592584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.592585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.592748 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:16.592825 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:16.593246 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 12:25:16.593334 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:25:16.594245 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.581581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36235 (local address 127.0.98.3:54305)
0620 12:25:16.582087 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:16.582130 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:16.583662 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:16.584246 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:16.584256 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:16.584808 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:16.585375 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.585391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.586314 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.586318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.587356 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.587366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.587892 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.587897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.588068 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.588627 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:16.588643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:16.590100 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:16.591692 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:16.591697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:16.591707 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:16.591967 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:16.592274 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:16.592277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:16.592278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:16.592482 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:16.592840 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:16.592844 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:16.593052 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 12:25:16.593203 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 12:25:16.593465 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":101,"threads_started":1}
I20260620 12:25:16.595353 4956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36235
I20260620 12:25:16.595539 4956 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:16.595907 4956 heartbeater.cc:511] Master 127.0.98.62:36235 requested a full tablet report, sending...
I20260620 12:25:16.596935 4496 ts_manager.cc:194] Registered new tserver with Master: 3e11dfbc56a84463acc62587824dca7d (127.0.98.3:34609)
I20260620 12:25:16.597429 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:25:16.597698 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54305
I20260620 12:25:16.605748 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.599489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.0.98.62:36235)
0620 12:25:16.599610 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:16.599615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:16.599788 (+ 173us) server_negotiation.cc:407] Connection header received
0620 12:25:16.599866 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:16.599868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:16.599901 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:16.599967 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:16.600610 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.601103 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.601852 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.602044 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.602115 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.603320 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:16.603322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:16.603324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:16.603344 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:16.604487 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.604849 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.604851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.604852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.604917 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:16.605144 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:16.605145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:16.605146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:16.605321 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:16.605416 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:16.605572 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 12:25:16.605651 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:25:16.608202 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:16.611166 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:16.625432 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.620209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.0.98.1:41371)
0620 12:25:16.620590 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:16.620597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:16.620619 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:25:16.620694 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:16.620699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:16.620898 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:16.621040 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:16.621355 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.622275 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.623920 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.624762 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.624917 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.625063 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:16.625191 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 12:25:16.625276 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":110,"threads_started":1}
I20260620 12:25:16.625425 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.619386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41371 (local address 127.0.0.1:40972)
0620 12:25:16.619733 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:16.619791 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:16.619916 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:16.621074 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:16.621081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:16.621101 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:16.621232 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.621239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.622406 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.622411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.623802 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.623812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.624904 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.624907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.625012 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.625047 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 12:25:16.625161 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 12:25:16.625276 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260620 12:25:16.625593 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.619440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.0.98.3:34609)
0620 12:25:16.620262 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:16.620273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:16.620505 (+ 232us) server_negotiation.cc:407] Connection header received
0620 12:25:16.620707 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:16.620712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:16.620898 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:16.621029 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:16.621464 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.622692 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.624087 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.624784 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.624886 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.625024 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:16.625345 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 12:25:16.625444 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":166,"threads_started":1}
I20260620 12:25:16.626703 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.619391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34609 (local address 127.0.0.1:41888)
0620 12:25:16.620417 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:16.620429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:16.620572 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:16.621225 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:16.621227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:16.621241 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:16.621348 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.621353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.622828 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.622831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.623976 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.623983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.625086 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.625089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.625177 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.625201 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:25:16.625774 (+ 573us) client_negotiation.cc:240] Negotiation successful
0620 12:25:16.626524 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":683,"thread_start_us":124,"threads_started":1}
I20260620 12:25:16.627013 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.620449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.0.98.2:39473)
0620 12:25:16.621179 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:16.621189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:16.621211 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:25:16.621279 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:16.621285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:16.621503 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:16.621664 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:16.621879 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.622950 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.624416 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.625178 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.625280 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.625404 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:16.626794 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0620 12:25:16.626881 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":115,"threads_started":1}
I20260620 12:25:16.627110 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:16.620072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39473 (local address 127.0.0.1:59464)
0620 12:25:16.620860 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:16.620872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:16.620939 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:16.621663 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:16.621666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:16.621683 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:16.621762 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.621767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.623087 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.623090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.624291 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:16.624302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:16.625459 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:16.625463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:16.626509 (+ 1046us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:16.626533 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:25:16.626605 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 12:25:16.626690 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":27,"thread_start_us":79,"threads_started":1}
I20260620 12:25:16.628593 4623 tablet_service.cc:1467] Processing CreateTablet for tablet 999324c9f4bb40d49203e02251f076a0 (DEFAULT_TABLE table=test-table [id=d65ecf57cec646f0962c82b1e72a5202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:16.629631 4623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999324c9f4bb40d49203e02251f076a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:16.629411 4757 tablet_service.cc:1467] Processing CreateTablet for tablet 999324c9f4bb40d49203e02251f076a0 (DEFAULT_TABLE table=test-table [id=d65ecf57cec646f0962c82b1e72a5202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:16.630430 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999324c9f4bb40d49203e02251f076a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:16.631497 4890 tablet_service.cc:1467] Processing CreateTablet for tablet 999324c9f4bb40d49203e02251f076a0 (DEFAULT_TABLE table=test-table [id=d65ecf57cec646f0962c82b1e72a5202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:16.632439 4890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999324c9f4bb40d49203e02251f076a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:16.636343 4974 tablet_bootstrap.cc:492] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Bootstrap starting.
I20260620 12:25:16.637308 4975 tablet_bootstrap.cc:492] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896: Bootstrap starting.
I20260620 12:25:16.638512 4974 tablet_bootstrap.cc:654] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:16.639441 4975 tablet_bootstrap.cc:654] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:16.639431 4976 tablet_bootstrap.cc:492] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Bootstrap starting.
I20260620 12:25:16.639750 4974 log.cc:824] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:16.640725 4975 log.cc:824] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:16.641819 4976 tablet_bootstrap.cc:654] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:16.642165 4974 tablet_bootstrap.cc:492] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: No bootstrap required, opened a new log
I20260620 12:25:16.642385 4974 ts_tablet_manager.cc:1397] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:25:16.642396 4975 tablet_bootstrap.cc:492] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896: No bootstrap required, opened a new log
I20260620 12:25:16.642580 4975 ts_tablet_manager.cc:1397] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:16.643105 4976 log.cc:824] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:16.644944 4976 tablet_bootstrap.cc:492] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: No bootstrap required, opened a new log
I20260620 12:25:16.645128 4976 ts_tablet_manager.cc:1397] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:25:16.645634 4975 raft_consensus.cc:348] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:16.645856 4975 raft_consensus.cc:374] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:16.645906 4975 raft_consensus.cc:729] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1795bbd0bc47aa961c2ff621b8a896, State: Initialized, Role: FOLLOWER
I20260620 12:25:16.646126 4974 raft_consensus.cc:348] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:16.646375 4974 raft_consensus.cc:374] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:16.646427 4974 raft_consensus.cc:729] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0bd64774b24b57a936269268a40029, State: Initialized, Role: FOLLOWER
I20260620 12:25:16.646366 4975 consensus_queue.cc:260] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:16.646867 4974 consensus_queue.cc:260] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:16.647352 4975 ts_tablet_manager.cc:1428] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:25:16.647677 4689 heartbeater.cc:503] Master 127.0.98.62:36235 was elected leader, sending a full tablet report...
I20260620 12:25:16.648022 4974 ts_tablet_manager.cc:1428] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:25:16.648352 4823 heartbeater.cc:503] Master 127.0.98.62:36235 was elected leader, sending a full tablet report...
I20260620 12:25:16.648824 4976 raft_consensus.cc:348] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:16.649063 4976 raft_consensus.cc:374] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:16.649281 4976 raft_consensus.cc:729] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e11dfbc56a84463acc62587824dca7d, State: Initialized, Role: FOLLOWER
I20260620 12:25:16.649704 4976 consensus_queue.cc:260] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:16.650326 4956 heartbeater.cc:503] Master 127.0.98.62:36235 was elected leader, sending a full tablet report...
I20260620 12:25:16.650599 4976 ts_tablet_manager.cc:1428] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 12:25:16.834896 4957 tablet.cc:2369] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:25:16.870352 4824 tablet.cc:2369] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:25:16.890865 4690 tablet.cc:2369] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:17.083137 4981 raft_consensus.cc:482] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:17.083379 4981 raft_consensus.cc:504] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:17.084524 4981 leader_election.cc:288] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e11dfbc56a84463acc62587824dca7d (127.0.98.3:34609), cf1795bbd0bc47aa961c2ff621b8a896 (127.0.98.1:41371)
I20260620 12:25:17.088621 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.084919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51061 (local address 127.0.98.1:41371)
0620 12:25:17.085054 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.085059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.085351 (+ 292us) server_negotiation.cc:407] Connection header received
0620 12:25:17.085482 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.085485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.085537 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.085620 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:17.085932 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.086492 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.087646 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.088056 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.088175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:17.088235 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.088421 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.088486 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:25:17.088640 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.084893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52479 (local address 127.0.98.3:34609)
0620 12:25:17.085032 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.085038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.085199 (+ 161us) server_negotiation.cc:407] Connection header received
0620 12:25:17.085317 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.085322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.085376 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.085453 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:17.085754 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.086267 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.087673 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.088076 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.088191 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:17.088251 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.088445 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.088514 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:17.088666 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.084776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34609 (local address 127.0.98.2:52479)
0620 12:25:17.085107 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:17.085119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:17.085216 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:17.085472 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:17.085475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:17.085504 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:17.085654 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:17.085661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.086378 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.086381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:17.087474 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:17.087483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.088204 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.088208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:17.088326 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:17.088343 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:25:17.088430 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:25:17.088514 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":44,"thread_start_us":72,"threads_started":1}
I20260620 12:25:17.088671 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.084780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41371 (local address 127.0.98.2:51061)
0620 12:25:17.085247 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:17.085264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:17.085360 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:17.085675 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:17.085678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:17.085694 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:17.085807 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:17.085814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.086609 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.086614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:17.087541 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:17.087553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.088189 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.088192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:17.088285 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:17.088313 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:25:17.088406 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 12:25:17.088479 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":59,"thread_start_us":61,"threads_started":1}
I20260620 12:25:17.089264 4643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999324c9f4bb40d49203e02251f076a0" candidate_uuid: "ad0bd64774b24b57a936269268a40029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" is_pre_election: true
I20260620 12:25:17.089255 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999324c9f4bb40d49203e02251f076a0" candidate_uuid: "ad0bd64774b24b57a936269268a40029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e11dfbc56a84463acc62587824dca7d" is_pre_election: true
I20260620 12:25:17.089555 4910 raft_consensus.cc:2454] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad0bd64774b24b57a936269268a40029 in term 0.
I20260620 12:25:17.089564 4643 raft_consensus.cc:2454] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad0bd64774b24b57a936269268a40029 in term 0.
I20260620 12:25:17.090018 4710 leader_election.cc:302] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e11dfbc56a84463acc62587824dca7d, ad0bd64774b24b57a936269268a40029; no voters:
I20260620 12:25:17.090287 4981 raft_consensus.cc:2781] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:17.090384 4981 raft_consensus.cc:482] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:17.090427 4981 raft_consensus.cc:3037] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:17.091570 4981 raft_consensus.cc:504] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:17.091902 4981 leader_election.cc:288] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [CANDIDATE]: Term 1 election: Requested vote from peers 3e11dfbc56a84463acc62587824dca7d (127.0.98.3:34609), cf1795bbd0bc47aa961c2ff621b8a896 (127.0.98.1:41371)
I20260620 12:25:17.092219 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999324c9f4bb40d49203e02251f076a0" candidate_uuid: "ad0bd64774b24b57a936269268a40029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e11dfbc56a84463acc62587824dca7d"
I20260620 12:25:17.092283 4643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999324c9f4bb40d49203e02251f076a0" candidate_uuid: "ad0bd64774b24b57a936269268a40029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1795bbd0bc47aa961c2ff621b8a896"
I20260620 12:25:17.092327 4910 raft_consensus.cc:3037] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:17.092438 4643 raft_consensus.cc:3037] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:17.093360 4910 raft_consensus.cc:2454] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad0bd64774b24b57a936269268a40029 in term 1.
I20260620 12:25:17.093501 4643 raft_consensus.cc:2454] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad0bd64774b24b57a936269268a40029 in term 1.
I20260620 12:25:17.093663 4710 leader_election.cc:302] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e11dfbc56a84463acc62587824dca7d, ad0bd64774b24b57a936269268a40029; no voters:
I20260620 12:25:17.093897 4981 raft_consensus.cc:2781] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:17.094136 4981 raft_consensus.cc:686] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 1 LEADER]: Becoming Leader. State: Replica: ad0bd64774b24b57a936269268a40029, State: Running, Role: LEADER
I20260620 12:25:17.094458 4981 consensus_queue.cc:237] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } }
I20260620 12:25:17.097635 4499 catalog_manager.cc:5495] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad0bd64774b24b57a936269268a40029 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ad0bd64774b24b57a936269268a40029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad0bd64774b24b57a936269268a40029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39473 } health_report { overall_health: HEALTHY } } }
I20260620 12:25:17.184019 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.180328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.98.2:39473)
0620 12:25:17.180554 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.180558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.180572 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:17.180641 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.180643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.180695 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.180790 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:17.181011 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.181506 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.182344 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:17.182578 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:17.182769 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:17.183004 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:17.183609 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:17.183708 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.183757 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.183804 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":59,"threads_started":1}
I20260620 12:25:17.190011 4643 raft_consensus.cc:1261] T 999324c9f4bb40d49203e02251f076a0 P cf1795bbd0bc47aa961c2ff621b8a896 [term 1 FOLLOWER]: Refusing update from remote peer ad0bd64774b24b57a936269268a40029: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:17.190011 4910 raft_consensus.cc:1261] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 1 FOLLOWER]: Refusing update from remote peer ad0bd64774b24b57a936269268a40029: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:17.190752 4981 consensus_queue.cc:1035] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e11dfbc56a84463acc62587824dca7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:17.190938 4985 consensus_queue.cc:1035] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1795bbd0bc47aa961c2ff621b8a896" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:17.197041 4992 mvcc.cc:205] Tried to move back new op lower bound from 7298901267202797568 to 7298901266822205440. Current Snapshot: MvccSnapshot[applied={T|T < 7298901267202797568}]
I20260620 12:25:17.206125 4499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:17.206269 4499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:17.208602 4499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c432108052f04771b9a54266a90061c8: Sending DeleteTablet for 3 replicas of tablet 999324c9f4bb40d49203e02251f076a0
I20260620 12:25:17.209318 4623 tablet_service.cc:1514] Processing DeleteTablet for tablet 999324c9f4bb40d49203e02251f076a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40972
I20260620 12:25:17.209339 4757 tablet_service.cc:1514] Processing DeleteTablet for tablet 999324c9f4bb40d49203e02251f076a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59464
I20260620 12:25:17.209781 4890 tablet_service.cc:1514] Processing DeleteTablet for tablet 999324c9f4bb40d49203e02251f076a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41888
I20260620 12:25:17.210160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 4561
I20260620 12:25:17.210295 5004 tablet_replica.cc:330] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: stopping tablet replica
I20260620 12:25:17.210602 5004 raft_consensus.cc:2229] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:17.210961 5004 raft_consensus.cc:2258] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:17.211371 5003 tablet_replica.cc:330] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: stopping tablet replica
I20260620 12:25:17.211622 5003 raft_consensus.cc:2229] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:17.211920 5003 raft_consensus.cc:2258] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:17.212946 5004 ts_tablet_manager.cc:1905] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:17.215260 5004 ts_tablet_manager.cc:1918] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:17.215369 5004 log.cc:1196] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/999324c9f4bb40d49203e02251f076a0
I20260620 12:25:17.215668 5004 ts_tablet_manager.cc:1939] T 999324c9f4bb40d49203e02251f076a0 P 3e11dfbc56a84463acc62587824dca7d: Deleting consensus metadata
I20260620 12:25:17.216725 4483 catalog_manager.cc:4841] TS 3e11dfbc56a84463acc62587824dca7d (127.0.98.3:34609): tablet 999324c9f4bb40d49203e02251f076a0 (table test-table [id=d65ecf57cec646f0962c82b1e72a5202]) successfully deleted
I20260620 12:25:17.217410 5003 ts_tablet_manager.cc:1905] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:17.219493 5003 ts_tablet_manager.cc:1918] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:17.219602 5003 log.cc:1196] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/999324c9f4bb40d49203e02251f076a0
I20260620 12:25:17.219944 5003 ts_tablet_manager.cc:1939] T 999324c9f4bb40d49203e02251f076a0 P ad0bd64774b24b57a936269268a40029: Deleting consensus metadata
W20260620 12:25:17.220153 4484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41371 (error 108)
I20260620 12:25:17.221033 4486 catalog_manager.cc:4841] TS ad0bd64774b24b57a936269268a40029 (127.0.98.2:39473): tablet 999324c9f4bb40d49203e02251f076a0 (table test-table [id=d65ecf57cec646f0962c82b1e72a5202]) successfully deleted
I20260620 12:25:17.221215 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.220811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41371 (local address 127.0.0.1:40974)
0620 12:25:17.221036 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:17.221111 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":55,"threads_started":1}
W20260620 12:25:17.221344 4484 catalog_manager.cc:4584] TS cf1795bbd0bc47aa961c2ff621b8a896 (127.0.98.1:41371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 999324c9f4bb40d49203e02251f076a0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41371: connect: Connection refused (error 111)
I20260620 12:25:17.221661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 4695
I20260620 12:25:17.227412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 4828
I20260620 12:25:17.232865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 4468
2026-06-20T12:25:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3175 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T12:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:17Z Disabled control of system clock
I20260620 12:25:17.262739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36991
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:25:17.369107 5010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:17.369356 5010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:17.369392 5010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:17.373400 5010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:17.373461 5010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:17.373476 5010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:17.373492 5010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:17.373507 5010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:17.378033 5010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:17.379022 5010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:17.380288 5010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:25:17.388597 5010 server_base.cc:1034] running on GCE node
W20260620 12:25:17.388592 5016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:17.388592 5015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:17.388571 5018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:17.389377 5010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:17.390378 5010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:17.391536 5010 hybrid_clock.cc:648] HybridClock initialized: now 1781958317391505 us; error 46 us; skew 500 ppm
I20260620 12:25:17.393330 5010 webserver.cc:460] Webserver started at http://127.0.98.62:38373/ using document root <none> and password file <none>
I20260620 12:25:17.393893 5010 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:17.393961 5010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:17.394196 5010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:17.396237 5010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "14da115fe5d84c2c898be292b873f3f5"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "2d7fd3d2b8bffc6fcfa63df957e71777"
server_key_iv: "2b86dd5ea583deb89c0a24c826a207d2"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.396780 5010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "14da115fe5d84c2c898be292b873f3f5"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "2d7fd3d2b8bffc6fcfa63df957e71777"
server_key_iv: "2b86dd5ea583deb89c0a24c826a207d2"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.400333 5010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 12:25:17.402638 5024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.403847 5010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 12:25:17.403972 5010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "14da115fe5d84c2c898be292b873f3f5"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "2d7fd3d2b8bffc6fcfa63df957e71777"
server_key_iv: "2b86dd5ea583deb89c0a24c826a207d2"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.404080 5010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:17.408576 5010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:17.409168 5010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:17.409350 5010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:17.416263 5010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36991
I20260620 12:25:17.416329 5076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36991 every 8 connection(s)
I20260620 12:25:17.417191 5010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:17.417670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5010
I20260620 12:25:17.417824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:17.418102 392 external_mini_cluster.cc:1428] Setting key 0755f9f89295d645e58c17d37dcd3d5d
I20260620 12:25:17.420490 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:17.424405 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.419511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.0.98.62:36991)
0620 12:25:17.420721 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.420736 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.420770 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:25:17.421394 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.421409 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.421734 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.422062 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:17.422959 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:17.422965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:17.422980 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:17.422993 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:17.423061 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:17.423086 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:17.423238 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.423304 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.423541 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":685,"threads_started":1}
I20260620 12:25:17.426811 5077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5: Bootstrap starting.
I20260620 12:25:17.428741 5077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:17.429535 5077 log.cc:824] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:17.431337 5077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5: No bootstrap required, opened a new log
I20260620 12:25:17.434691 5077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14da115fe5d84c2c898be292b873f3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36991 } }
I20260620 12:25:17.434904 5077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:17.434984 5077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14da115fe5d84c2c898be292b873f3f5, State: Initialized, Role: FOLLOWER
I20260620 12:25:17.435371 5077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14da115fe5d84c2c898be292b873f3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36991 } }
I20260620 12:25:17.435518 5077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:17.435582 5077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:17.435689 5077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:17.436493 5077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14da115fe5d84c2c898be292b873f3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36991 } }
I20260620 12:25:17.436851 5077 leader_election.cc:302] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14da115fe5d84c2c898be292b873f3f5; no voters:
I20260620 12:25:17.437175 5077 leader_election.cc:288] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:17.437278 5082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:17.437522 5082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [term 1 LEADER]: Becoming Leader. State: Replica: 14da115fe5d84c2c898be292b873f3f5, State: Running, Role: LEADER
I20260620 12:25:17.437811 5082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14da115fe5d84c2c898be292b873f3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36991 } }
I20260620 12:25:17.438334 5077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:17.439410 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14da115fe5d84c2c898be292b873f3f5. Latest consensus state: current_term: 1 leader_uuid: "14da115fe5d84c2c898be292b873f3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14da115fe5d84c2c898be292b873f3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36991 } } }
I20260620 12:25:17.439538 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:17.439796 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14da115fe5d84c2c898be292b873f3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14da115fe5d84c2c898be292b873f3f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36991 } } }
I20260620 12:25:17.439872 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:17.439899 5091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:17.443126 5091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:17.447865 5091 catalog_manager.cc:1347] Generated new cluster ID: 7ca7535a43e04b99bad528ea81a99273
I20260620 12:25:17.447924 5091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:17.459175 5091 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:17.459472 5091 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:17.464712 5091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5: Generated new TSK 0
I20260620 12:25:17.465299 5091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:17.472388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36991
--builtin_ntp_servers=127.0.98.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:25:17.578361 5101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:17.578639 5101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:17.578732 5101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:17.582664 5101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:17.582741 5101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:17.582830 5101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:17.587553 5101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:17.588557 5101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:17.589879 5101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:17.599304 5106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:17.599359 5109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:17.599315 5107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:17.599388 5101 server_base.cc:1034] running on GCE node
I20260620 12:25:17.599910 5101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:17.600557 5101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:17.601729 5101 hybrid_clock.cc:648] HybridClock initialized: now 1781958317601692 us; error 43 us; skew 500 ppm
I20260620 12:25:17.603533 5101 webserver.cc:460] Webserver started at http://127.0.98.1:38927/ using document root <none> and password file <none>
I20260620 12:25:17.604038 5101 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:17.604094 5101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:17.604249 5101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:17.606144 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "36d2577e2ef542da952beebcdccc7aa1"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "912e7db77e7645d5d6ca25b7916df531"
server_key_iv: "7e517b7e1beaf8b501c40e0fdf2e4079"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.606701 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "36d2577e2ef542da952beebcdccc7aa1"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "912e7db77e7645d5d6ca25b7916df531"
server_key_iv: "7e517b7e1beaf8b501c40e0fdf2e4079"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.610339 5101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:17.612763 5115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.613994 5101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:17.614084 5101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "36d2577e2ef542da952beebcdccc7aa1"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "912e7db77e7645d5d6ca25b7916df531"
server_key_iv: "7e517b7e1beaf8b501c40e0fdf2e4079"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.614166 5101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:17.618531 5101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:17.619110 5101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:17.619243 5101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:17.619704 5101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:17.620503 5101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:17.620548 5101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.620582 5101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:17.620599 5101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.629066 5101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42003
I20260620 12:25:17.629128 5228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42003 every 8 connection(s)
I20260620 12:25:17.629937 5101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:17.635686 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.631720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41425 (local address 127.0.98.62:36991)
0620 12:25:17.631873 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.631877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.632876 (+ 999us) server_negotiation.cc:407] Connection header received
0620 12:25:17.634125 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.634127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.634162 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.634234 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:17.635315 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:17.635318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:17.635320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:17.635331 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:17.635350 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:17.635359 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:17.635433 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.635551 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.635585 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:25:17.636564 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.632086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36991 (local address 127.0.98.1:41425)
0620 12:25:17.632731 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:17.632768 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:17.633949 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:17.634420 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:17.634426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:17.634508 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:17.635084 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:17.635086 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:17.635122 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:17.635183 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:17.635435 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:17.635441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:17.635446 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:25:17.635616 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 12:25:17.635841 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260620 12:25:17.637794 5229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36991
I20260620 12:25:17.638025 5229 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:17.638320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5101
I20260620 12:25:17.638434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:17.638477 5229 heartbeater.cc:511] Master 127.0.98.62:36991 requested a full tablet report, sending...
I20260620 12:25:17.638785 392 external_mini_cluster.cc:1428] Setting key bb04579d545c6ffffce00f9dbb47df1b
I20260620 12:25:17.640149 5041 ts_manager.cc:194] Registered new tserver with Master: 36d2577e2ef542da952beebcdccc7aa1 (127.0.98.1:42003)
I20260620 12:25:17.641250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36991
--builtin_ntp_servers=127.0.98.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:25:17.748144 5232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:17.748378 5232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:17.748417 5232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:17.752374 5232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:17.752419 5232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:17.752513 5232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:17.757088 5232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:17.758117 5232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:17.759485 5232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:17.768898 5238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:17.768922 5240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:17.768903 5237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:17.769151 5232 server_base.cc:1034] running on GCE node
I20260620 12:25:17.769675 5232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:17.770303 5232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:17.771481 5232 hybrid_clock.cc:648] HybridClock initialized: now 1781958317771435 us; error 58 us; skew 500 ppm
I20260620 12:25:17.773307 5232 webserver.cc:460] Webserver started at http://127.0.98.2:40897/ using document root <none> and password file <none>
I20260620 12:25:17.773890 5232 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:17.773980 5232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:17.774202 5232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:17.776279 5232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "19745fcf79a34fd1be8f3de4b39372fe"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "f37db705a5565188d3b927a37b6edf7d"
server_key_iv: "4071a4b794fc585ac62f15c7a195d12c"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.776854 5232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "19745fcf79a34fd1be8f3de4b39372fe"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "f37db705a5565188d3b927a37b6edf7d"
server_key_iv: "4071a4b794fc585ac62f15c7a195d12c"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.780447 5232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 12:25:17.782747 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.783855 5232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:17.783985 5232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "19745fcf79a34fd1be8f3de4b39372fe"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "f37db705a5565188d3b927a37b6edf7d"
server_key_iv: "4071a4b794fc585ac62f15c7a195d12c"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.784094 5232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:17.788443 5232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:17.789008 5232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:17.789182 5232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:17.789693 5232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:17.790537 5232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:17.790607 5232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.790705 5232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:17.790750 5232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.799737 5232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33309
I20260620 12:25:17.799757 5359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33309 every 8 connection(s)
I20260620 12:25:17.800576 5232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:17.806707 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.802344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36201 (local address 127.0.98.62:36991)
0620 12:25:17.802502 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.802505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.803607 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 12:25:17.804987 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.804991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.805035 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.805142 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:17.806214 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:17.806218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:17.806220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:17.806240 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:17.806260 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:17.806268 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:17.806369 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.806511 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.806555 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:25:17.807111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5232
I20260620 12:25:17.807224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:17.807523 392 external_mini_cluster.cc:1428] Setting key d9579d2f8f7c7ba2f9930d895144f557
I20260620 12:25:17.808010 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.802792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36991 (local address 127.0.98.2:36201)
0620 12:25:17.803450 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:17.803489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:17.804785 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:17.805305 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:17.805312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:17.805396 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:17.805927 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:17.805928 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:17.805967 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:17.806032 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:17.806353 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:17.806361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:17.806366 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:25:17.806550 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 12:25:17.806814 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260620 12:25:17.809051 5360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36991
I20260620 12:25:17.809266 5360 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:17.809710 5360 heartbeater.cc:511] Master 127.0.98.62:36991 requested a full tablet report, sending...
I20260620 12:25:17.809798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36991
--builtin_ntp_servers=127.0.98.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 12:25:17.810838 5041 ts_manager.cc:194] Registered new tserver with Master: 19745fcf79a34fd1be8f3de4b39372fe (127.0.98.2:33309)
W20260620 12:25:17.919373 5363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:17.919762 5363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:17.919845 5363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:17.924634 5363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:17.924685 5363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:17.924741 5363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:17.929495 5363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:17.930527 5363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:17.931890 5363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:17.941407 5368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:17.941437 5371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:17.941606 5363 server_base.cc:1034] running on GCE node
W20260620 12:25:17.941767 5369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:17.942143 5363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:17.942793 5363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:17.943945 5363 hybrid_clock.cc:648] HybridClock initialized: now 1781958317943915 us; error 43 us; skew 500 ppm
I20260620 12:25:17.945981 5363 webserver.cc:460] Webserver started at http://127.0.98.3:44039/ using document root <none> and password file <none>
I20260620 12:25:17.946556 5363 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:17.946615 5363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:17.946861 5363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:17.948822 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "b88e6568db5d4eb4a3b8d027952add43"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "032983ebd95bef2f6697fe1bd44e8566"
server_key_iv: "e38ff42f48a9715d90c4c2db1861476d"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.949370 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b88e6568db5d4eb4a3b8d027952add43"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "032983ebd95bef2f6697fe1bd44e8566"
server_key_iv: "e38ff42f48a9715d90c4c2db1861476d"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.953374 5363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 12:25:17.955788 5377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.956933 5363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:17.957063 5363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "b88e6568db5d4eb4a3b8d027952add43"
format_stamp: "Formatted at 2026-06-20 12:25:17 on dist-test-slave-bwpp"
server_key: "032983ebd95bef2f6697fe1bd44e8566"
server_key_iv: "e38ff42f48a9715d90c4c2db1861476d"
server_key_version: "encryptionkey@0"
I20260620 12:25:17.957173 5363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:17.961630 5363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:17.962183 5363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:17.962376 5363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:17.962982 5363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:17.963816 5363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:17.963891 5363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.963979 5363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:17.964031 5363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:17.972898 5363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37237
I20260620 12:25:17.972916 5490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37237 every 8 connection(s)
I20260620 12:25:17.973763 5363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:17.975785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5363
I20260620 12:25:17.975914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:17.976204 392 external_mini_cluster.cc:1428] Setting key 2903a9c1f371c5054cbdd431fe64af4c
I20260620 12:25:17.980386 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.975562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42381 (local address 127.0.98.62:36991)
0620 12:25:17.975722 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.975728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.976938 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 12:25:17.978720 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.978725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.978782 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.978885 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:17.980058 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:17.980061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:17.980063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:17.980074 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:17.980092 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:17.980100 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:17.980168 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.980245 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.980278 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:25:17.981242 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.976071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36991 (local address 127.0.98.3:42381)
0620 12:25:17.976785 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:17.976835 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:17.978400 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:17.979085 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:17.979092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:17.979181 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:17.979816 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:17.979818 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:17.979856 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:17.979922 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:17.980157 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:17.980161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:17.980164 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:25:17.980285 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 12:25:17.980474 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":144,"threads_started":1}
I20260620 12:25:17.982254 5491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36991
I20260620 12:25:17.982437 5491 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:17.982844 5491 heartbeater.cc:511] Master 127.0.98.62:36991 requested a full tablet report, sending...
I20260620 12:25:17.983757 5041 ts_manager.cc:194] Registered new tserver with Master: b88e6568db5d4eb4a3b8d027952add43 (127.0.98.3:37237)
I20260620 12:25:17.989988 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:25:17.993296 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:17.991892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.0.98.62:36991)
0620 12:25:17.992049 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:17.992052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:17.992142 (+ 90us) server_negotiation.cc:407] Connection header received
0620 12:25:17.992235 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:17.992237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:17.992263 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:17.992324 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:17.992915 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:17.992918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:17.992921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:17.992933 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:17.992949 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:17.992956 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:17.993020 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:17.993101 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 12:25:17.993130 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:25:17.995791 5041 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:17.998620 5041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:18.009742 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.007221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33309 (local address 127.0.0.1:50242)
0620 12:25:18.007576 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.007590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:18.007963 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:18.008368 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:18.008374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:18.008447 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:18.008976 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:18.008978 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:18.008997 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:18.009065 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:18.009397 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:18.009409 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:18.009417 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:25:18.009533 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 12:25:18.009604 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"spinlock_wait_cycles":658304,"thread_start_us":122,"threads_started":1}
I20260620 12:25:18.010022 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.007492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.0.98.2:33309)
0620 12:25:18.007871 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.007880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.008052 (+ 172us) server_negotiation.cc:407] Connection header received
0620 12:25:18.008100 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.008103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.008238 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.008388 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.009193 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.009198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.009207 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.009216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.009253 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.009302 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.009410 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.009564 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.009664 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260620 12:25:18.010583 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.006616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:50814)
0620 12:25:18.007462 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.007476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:18.007938 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:18.008788 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:18.008790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:18.008805 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:18.009916 (+ 1111us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:18.009919 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:18.009929 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:18.009951 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:18.010340 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:18.010343 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:18.010347 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:25:18.010422 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:25:18.010456 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":118,"threads_started":1}
I20260620 12:25:18.010879 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.006539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37237 (local address 127.0.0.1:48764)
0620 12:25:18.007200 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.007287 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:18.007384 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:18.009433 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:18.009436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:18.009449 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:18.009818 (+ 369us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:18.009820 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:18.009832 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:18.009858 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:18.010634 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:18.010637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:18.010641 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:25:18.010724 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 12:25:18.010759 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":51,"threads_started":1}
I20260620 12:25:18.011119 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.006993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.0.98.1:42003)
0620 12:25:18.008418 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.008431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.008452 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:25:18.008500 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.008504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.008664 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.008791 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.010092 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.010097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.010110 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.010125 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.010168 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.010201 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.010297 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.010862 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.010934 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":124,"threads_started":1}
I20260620 12:25:18.011417 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.006675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.98.3:37237)
0620 12:25:18.007930 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.007943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.007968 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:25:18.008023 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.008029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.008233 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.009641 (+ 1408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.010454 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.010458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.010468 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.010477 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.010512 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.010532 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.011127 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.011176 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.011242 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":142,"threads_started":1}
I20260620 12:25:18.013170 5293 tablet_service.cc:1467] Processing CreateTablet for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165 (DEFAULT_TABLE table=test-table [id=bc70a770a57041ba932920c810ef8e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:18.013578 5425 tablet_service.cc:1467] Processing CreateTablet for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165 (DEFAULT_TABLE table=test-table [id=bc70a770a57041ba932920c810ef8e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:18.013701 5163 tablet_service.cc:1467] Processing CreateTablet for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165 (DEFAULT_TABLE table=test-table [id=bc70a770a57041ba932920c810ef8e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:18.014281 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:18.014568 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:18.014628 5163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:18.020430 5508 tablet_bootstrap.cc:492] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Bootstrap starting.
I20260620 12:25:18.021879 5509 tablet_bootstrap.cc:492] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1: Bootstrap starting.
I20260620 12:25:18.022714 5508 tablet_bootstrap.cc:654] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:18.022987 5510 tablet_bootstrap.cc:492] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: Bootstrap starting.
I20260620 12:25:18.023471 5508 log.cc:824] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:18.024021 5509 tablet_bootstrap.cc:654] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:18.024776 5509 log.cc:824] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:18.024961 5510 tablet_bootstrap.cc:654] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:18.025269 5508 tablet_bootstrap.cc:492] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: No bootstrap required, opened a new log
I20260620 12:25:18.025473 5508 ts_tablet_manager.cc:1397] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:18.025724 5510 log.cc:824] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:18.026589 5509 tablet_bootstrap.cc:492] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1: No bootstrap required, opened a new log
I20260620 12:25:18.026788 5509 ts_tablet_manager.cc:1397] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 12:25:18.027575 5510 tablet_bootstrap.cc:492] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: No bootstrap required, opened a new log
I20260620 12:25:18.027741 5510 ts_tablet_manager.cc:1397] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:18.029333 5508 raft_consensus.cc:348] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.029562 5508 raft_consensus.cc:374] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:18.029614 5508 raft_consensus.cc:729] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88e6568db5d4eb4a3b8d027952add43, State: Initialized, Role: FOLLOWER
I20260620 12:25:18.030066 5508 consensus_queue.cc:260] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.030731 5491 heartbeater.cc:503] Master 127.0.98.62:36991 was elected leader, sending a full tablet report...
I20260620 12:25:18.031059 5508 ts_tablet_manager.cc:1428] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:25:18.031884 5509 raft_consensus.cc:348] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.032145 5509 raft_consensus.cc:374] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:18.032004 5510 raft_consensus.cc:348] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.032200 5509 raft_consensus.cc:729] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d2577e2ef542da952beebcdccc7aa1, State: Initialized, Role: FOLLOWER
I20260620 12:25:18.032220 5510 raft_consensus.cc:374] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:18.032284 5510 raft_consensus.cc:729] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19745fcf79a34fd1be8f3de4b39372fe, State: Initialized, Role: FOLLOWER
I20260620 12:25:18.032589 5509 consensus_queue.cc:260] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.032667 5510 consensus_queue.cc:260] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.033210 5229 heartbeater.cc:503] Master 127.0.98.62:36991 was elected leader, sending a full tablet report...
I20260620 12:25:18.033512 5509 ts_tablet_manager.cc:1428] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260620 12:25:18.033563 5510 ts_tablet_manager.cc:1428] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 12:25:18.033861 5360 heartbeater.cc:503] Master 127.0.98.62:36991 was elected leader, sending a full tablet report...
W20260620 12:25:18.051492 5361 tablet.cc:2369] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:25:18.131089 5230 tablet.cc:2369] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:25:18.224669 5492 tablet.cc:2369] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:18.284647 5516 raft_consensus.cc:482] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:18.284925 5516 raft_consensus.cc:504] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.286103 5516 leader_election.cc:288] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88e6568db5d4eb4a3b8d027952add43 (127.0.98.3:37237), 36d2577e2ef542da952beebcdccc7aa1 (127.0.98.1:42003)
I20260620 12:25:18.288208 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.286395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.98.2:42843)
0620 12:25:18.286552 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.286570 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:18.286690 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:18.286945 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:18.286947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:18.286960 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:18.287560 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:18.287564 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:18.287581 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:18.287616 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:18.287918 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:18.287921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:18.287924 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:25:18.288031 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 12:25:18.288069 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:25:18.288224 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.286506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42843 (local address 127.0.98.1:42003)
0620 12:25:18.286685 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.286690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.286710 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:25:18.286791 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.286795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.286834 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.286917 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.287757 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.287761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.287765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.287778 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.287802 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.287826 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.287920 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.288013 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.288080 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:25:18.288324 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.286680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51813 (local address 127.0.98.3:37237)
0620 12:25:18.286809 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.286814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.287001 (+ 187us) server_negotiation.cc:407] Connection header received
0620 12:25:18.287094 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.287100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.287142 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.287226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.287917 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.287921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.287923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.287936 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.287960 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.287975 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.288066 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.288159 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.288194 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:25:18.288383 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.286395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37237 (local address 127.0.98.2:51813)
0620 12:25:18.286911 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.286925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:18.287025 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:18.287239 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:18.287242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:18.287254 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:18.287740 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:18.287741 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:18.287752 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:18.287778 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:18.288066 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:18.288069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:18.288072 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:25:18.288148 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 12:25:18.288178 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":136,"thread_start_us":97,"threads_started":1}
I20260620 12:25:18.288944 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d85cfd98ab4f7ea5fc5b91b95bf165" candidate_uuid: "19745fcf79a34fd1be8f3de4b39372fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88e6568db5d4eb4a3b8d027952add43" is_pre_election: true
I20260620 12:25:18.288964 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d85cfd98ab4f7ea5fc5b91b95bf165" candidate_uuid: "19745fcf79a34fd1be8f3de4b39372fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d2577e2ef542da952beebcdccc7aa1" is_pre_election: true
I20260620 12:25:18.289238 5183 raft_consensus.cc:2454] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19745fcf79a34fd1be8f3de4b39372fe in term 0.
I20260620 12:25:18.289249 5445 raft_consensus.cc:2454] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19745fcf79a34fd1be8f3de4b39372fe in term 0.
I20260620 12:25:18.289763 5247 leader_election.cc:302] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19745fcf79a34fd1be8f3de4b39372fe, 36d2577e2ef542da952beebcdccc7aa1; no voters:
I20260620 12:25:18.290053 5516 raft_consensus.cc:2781] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:18.290138 5516 raft_consensus.cc:482] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:18.290179 5516 raft_consensus.cc:3037] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:18.291330 5516 raft_consensus.cc:504] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.291649 5516 leader_election.cc:288] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [CANDIDATE]: Term 1 election: Requested vote from peers b88e6568db5d4eb4a3b8d027952add43 (127.0.98.3:37237), 36d2577e2ef542da952beebcdccc7aa1 (127.0.98.1:42003)
I20260620 12:25:18.292035 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d85cfd98ab4f7ea5fc5b91b95bf165" candidate_uuid: "19745fcf79a34fd1be8f3de4b39372fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88e6568db5d4eb4a3b8d027952add43"
I20260620 12:25:18.292035 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d85cfd98ab4f7ea5fc5b91b95bf165" candidate_uuid: "19745fcf79a34fd1be8f3de4b39372fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d2577e2ef542da952beebcdccc7aa1"
I20260620 12:25:18.292165 5183 raft_consensus.cc:3037] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:18.292165 5445 raft_consensus.cc:3037] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:18.293438 5445 raft_consensus.cc:2454] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19745fcf79a34fd1be8f3de4b39372fe in term 1.
I20260620 12:25:18.293438 5183 raft_consensus.cc:2454] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19745fcf79a34fd1be8f3de4b39372fe in term 1.
I20260620 12:25:18.293715 5250 leader_election.cc:302] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19745fcf79a34fd1be8f3de4b39372fe, b88e6568db5d4eb4a3b8d027952add43; no voters:
I20260620 12:25:18.293890 5516 raft_consensus.cc:2781] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:18.294113 5516 raft_consensus.cc:686] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 1 LEADER]: Becoming Leader. State: Replica: 19745fcf79a34fd1be8f3de4b39372fe, State: Running, Role: LEADER
I20260620 12:25:18.294441 5516 consensus_queue.cc:237] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260620 12:25:18.297490 5040 catalog_manager.cc:5495] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 19745fcf79a34fd1be8f3de4b39372fe (127.0.98.2). New cstate: current_term: 1 leader_uuid: "19745fcf79a34fd1be8f3de4b39372fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19745fcf79a34fd1be8f3de4b39372fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } health_report { overall_health: HEALTHY } } }
I20260620 12:25:18.370967 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.369303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.0.98.2:33309)
0620 12:25:18.369475 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.369480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.369495 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:18.369654 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.369656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.369718 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.369828 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.370479 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.370483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.370486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.370497 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.370516 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.370528 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.370630 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.370764 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.370811 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 12:25:18.376909 5445 raft_consensus.cc:1261] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 1 FOLLOWER]: Refusing update from remote peer 19745fcf79a34fd1be8f3de4b39372fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:18.376924 5183 raft_consensus.cc:1261] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 36d2577e2ef542da952beebcdccc7aa1 [term 1 FOLLOWER]: Refusing update from remote peer 19745fcf79a34fd1be8f3de4b39372fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:18.377640 5518 consensus_queue.cc:1035] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d2577e2ef542da952beebcdccc7aa1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:18.377812 5516 consensus_queue.cc:1035] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88e6568db5d4eb4a3b8d027952add43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:18.386358 5522 mvcc.cc:205] Tried to move back new op lower bound from 7298901272064827392 to 7298901271736803328. Current Snapshot: MvccSnapshot[applied={T|T < 7298901272064827392}]
I20260620 12:25:18.395639 5039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:18.395807 5039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:18.398123 5039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14da115fe5d84c2c898be292b873f3f5: Sending DeleteTablet for 3 replicas of tablet f8d85cfd98ab4f7ea5fc5b91b95bf165
I20260620 12:25:18.398881 5163 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:18 UTC) from {username='slave'} at 127.0.0.1:50814
I20260620 12:25:18.398911 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:18 UTC) from {username='slave'} at 127.0.0.1:50242
I20260620 12:25:18.398991 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:18 UTC) from {username='slave'} at 127.0.0.1:48764
I20260620 12:25:18.399333 5531 tablet_replica.cc:330] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe: stopping tablet replica
I20260620 12:25:18.399621 5531 raft_consensus.cc:2229] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:18.399663 5533 tablet_replica.cc:330] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: stopping tablet replica
I20260620 12:25:18.400000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5101
I20260620 12:25:18.399994 5531 raft_consensus.cc:2258] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P 19745fcf79a34fd1be8f3de4b39372fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:18.400030 5533 raft_consensus.cc:2229] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:18.400424 5533 raft_consensus.cc:2258] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:18.404222 5533 ts_tablet_manager.cc:1905] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 12:25:18.406738 5025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42003 (error 108)
I20260620 12:25:18.406939 5533 ts_tablet_manager.cc:1918] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:18.407037 5533 log.cc:1196] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/f8d85cfd98ab4f7ea5fc5b91b95bf165
I20260620 12:25:18.407389 5533 ts_tablet_manager.cc:1939] T f8d85cfd98ab4f7ea5fc5b91b95bf165 P b88e6568db5d4eb4a3b8d027952add43: Deleting consensus metadata
I20260620 12:25:18.407514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5232
I20260620 12:25:18.409826 5028 catalog_manager.cc:4841] TS b88e6568db5d4eb4a3b8d027952add43 (127.0.98.3:37237): tablet f8d85cfd98ab4f7ea5fc5b91b95bf165 (table test-table [id=bc70a770a57041ba932920c810ef8e6a]) successfully deleted
I20260620 12:25:18.409827 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.409463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:50822)
0620 12:25:18.409638 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.409723 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 12:25:18.410118 5025 catalog_manager.cc:4584] TS 36d2577e2ef542da952beebcdccc7aa1 (127.0.98.1:42003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8d85cfd98ab4f7ea5fc5b91b95bf165: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
I20260620 12:25:18.413170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5363
I20260620 12:25:18.414948 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.413374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33309 (local address 127.0.0.1:50264)
0620 12:25:18.414830 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.414885 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1362}
I20260620 12:25:18.418853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5010
2026-06-20T12:25:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1185 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 12:25:18.440014 392 mini_oidc.cc:152] Starting JWKS server
I20260620 12:25:18.441455 392 webserver.cc:460] Webserver started at https://127.0.0.1:45819/ using document root <none> and password file <none>
I20260620 12:25:18.441582 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:25:18.441797 392 webserver.cc:460] Webserver started at http://127.0.0.1:34975/ using document root <none> and password file <none>
2026-06-20T12:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:18Z Disabled control of system clock
I20260620 12:25:18.449929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34859
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45819/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/testchainca.cert with env {}
W20260620 12:25:18.554767 5540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:18.555043 5540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:18.555114 5540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:18.559064 5540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:18.559142 5540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:18.559170 5540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:18.559206 5540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:18.559238 5540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:25:18.559296 5540 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:25:18.559336 5540 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45819/jwks.json
I20260620 12:25:18.564647 5540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45819/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:18.565618 5540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:18.566700 5540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:18.574779 5548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:18.574801 5545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:18.574784 5546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:18.574995 5540 server_base.cc:1034] running on GCE node
I20260620 12:25:18.575567 5540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:18.576536 5540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:18.577692 5540 hybrid_clock.cc:648] HybridClock initialized: now 1781958318577656 us; error 49 us; skew 500 ppm
I20260620 12:25:18.579457 5540 webserver.cc:460] Webserver started at http://127.0.98.62:41823/ using document root <none> and password file <none>
I20260620 12:25:18.580026 5540 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:18.580099 5540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:18.580318 5540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:18.582229 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "647d8bac2c644efe8918d03c00460323"
format_stamp: "Formatted at 2026-06-20 12:25:18 on dist-test-slave-bwpp"
server_key: "ae3df5299159d84278188185572dc725"
server_key_iv: "5d80a46a043e9c684d56d8dd281d0edb"
server_key_version: "encryptionkey@0"
I20260620 12:25:18.582823 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "647d8bac2c644efe8918d03c00460323"
format_stamp: "Formatted at 2026-06-20 12:25:18 on dist-test-slave-bwpp"
server_key: "ae3df5299159d84278188185572dc725"
server_key_iv: "5d80a46a043e9c684d56d8dd281d0edb"
server_key_version: "encryptionkey@0"
I20260620 12:25:18.586275 5540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:25:18.588702 5554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:18.589906 5540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:25:18.590051 5540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "647d8bac2c644efe8918d03c00460323"
format_stamp: "Formatted at 2026-06-20 12:25:18 on dist-test-slave-bwpp"
server_key: "ae3df5299159d84278188185572dc725"
server_key_iv: "5d80a46a043e9c684d56d8dd281d0edb"
server_key_version: "encryptionkey@0"
I20260620 12:25:18.590157 5540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:18.607988 5540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:18.608644 5540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:18.608845 5540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:18.615469 5540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34859
I20260620 12:25:18.615511 5606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34859 every 8 connection(s)
I20260620 12:25:18.616418 5540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:18.619619 5607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:18.624804 5607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323: Bootstrap starting.
I20260620 12:25:18.625159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5540
I20260620 12:25:18.625278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:18.625525 392 external_mini_cluster.cc:1428] Setting key 8417df03bb73f2685232abaf7d07ed0f
I20260620 12:25:18.627343 5607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:18.628034 5607 log.cc:824] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:18.629794 5607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323: No bootstrap required, opened a new log
I20260620 12:25:18.633270 5607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d8bac2c644efe8918d03c00460323" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260620 12:25:18.633502 5607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:18.633570 5607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647d8bac2c644efe8918d03c00460323, State: Initialized, Role: FOLLOWER
I20260620 12:25:18.634006 5607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d8bac2c644efe8918d03c00460323" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260620 12:25:18.634155 5607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:18.634236 5607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:18.634346 5607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:18.634563 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.626752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.98.62:34859)
0620 12:25:18.627356 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.627373 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.627434 (+ 61us) server_negotiation.cc:407] Connection header received
0620 12:25:18.628046 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.628062 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.628472 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.628784 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.629215 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.630224 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.631197 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.631652 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.631813 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:18.632712 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.632717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.632728 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.633109 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.633181 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.633200 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.633310 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.633455 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.633741 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260620 12:25:18.635383 5607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d8bac2c644efe8918d03c00460323" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260620 12:25:18.635710 5607 leader_election.cc:302] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 647d8bac2c644efe8918d03c00460323; no voters:
I20260620 12:25:18.636044 5607 leader_election.cc:288] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:18.636130 5612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:18.636596 5612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [term 1 LEADER]: Becoming Leader. State: Replica: 647d8bac2c644efe8918d03c00460323, State: Running, Role: LEADER
I20260620 12:25:18.637019 5612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d8bac2c644efe8918d03c00460323" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260620 12:25:18.637249 5607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:18.637334 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:25:18.638711 5613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "647d8bac2c644efe8918d03c00460323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d8bac2c644efe8918d03c00460323" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260620 12:25:18.638839 5613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:18.639139 5614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 647d8bac2c644efe8918d03c00460323. Latest consensus state: current_term: 1 leader_uuid: "647d8bac2c644efe8918d03c00460323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d8bac2c644efe8918d03c00460323" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260620 12:25:18.639227 5614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323 [sys.catalog]: This master's current role is: LEADER
W20260620 12:25:18.642220 5627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:25:18.642297 5627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:25:18.642367 5623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:18.643292 5623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:18.648519 5623 catalog_manager.cc:1347] Generated new cluster ID: 09994b59e1bd40d780754ae8593ccea2
I20260620 12:25:18.648598 5623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:18.658509 5623 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:18.659389 5623 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:18.670509 5623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 647d8bac2c644efe8918d03c00460323: Generated new TSK 0
I20260620 12:25:18.671182 5623 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 12:25:18.710767 5637 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 12:25:18.710880 5637 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 12:25:18.711359 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.687116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.0.98.62:34859)
0620 12:25:18.687253 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.687259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.687510 (+ 251us) server_negotiation.cc:407] Connection header received
0620 12:25:18.687682 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.687684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.687729 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.687822 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:25:18.688088 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.688607 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.689852 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.690099 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.690184 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:18.690618 (+ 434us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:25:18.710968 (+ 20350us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 12:25:18.711218 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 12:25:18.711457 5610 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 12:25:18.712708 5636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:18.687047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.0.1:39804)
0620 12:25:18.687395 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.687410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:18.687537 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:18.687846 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:18.687849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:18.687857 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:25:18.687978 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:18.687985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.688721 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.688725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:18.689715 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:18.689726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.690200 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.690235 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:18.690373 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:18.690422 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:25:18.711203 (+ 20781us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 12:25:18.711432 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":123,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 12:25:18.713620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5540
2026-06-20T12:25:18Z chronyd exiting
I20260620 12:25:18.751004 392 test_util.cc:175] -----------------------------------------------
I20260620 12:25:18.751108 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781958290914605-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 12:25:18.752179 392 mini_oidc.cc:152] Starting JWKS server
I20260620 12:25:18.753036 392 webserver.cc:460] Webserver started at https://127.0.0.1:36319/ using document root <none> and password file <none>
I20260620 12:25:18.753165 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:25:18.753444 392 webserver.cc:460] Webserver started at http://127.0.0.1:37017/ using document root <none> and password file <none>
2026-06-20T12:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:18Z Disabled control of system clock
I20260620 12:25:18.762007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38683
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38683
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36319/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/testchainca.cert with env {}
W20260620 12:25:18.869474 5643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:18.869714 5643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:18.869751 5643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:18.873728 5643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:18.873778 5643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:18.873795 5643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:18.873811 5643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:18.873826 5643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:25:18.873842 5643 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:25:18.873854 5643 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36319/jwks.json
I20260620 12:25:18.879079 5643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36319/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:18.879982 5643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:18.880921 5643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:18.889295 5648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:18.889367 5649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:18.889423 5643 server_base.cc:1034] running on GCE node
W20260620 12:25:18.889362 5651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:18.890158 5643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:18.891335 5643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:18.892535 5643 hybrid_clock.cc:648] HybridClock initialized: now 1781958318892508 us; error 39 us; skew 500 ppm
I20260620 12:25:18.894423 5643 webserver.cc:460] Webserver started at http://127.0.98.62:44401/ using document root <none> and password file <none>
I20260620 12:25:18.895078 5643 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:18.895175 5643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:18.895411 5643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:18.897476 5643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "297b60a9d72c400f926b8c145525bd0e"
format_stamp: "Formatted at 2026-06-20 12:25:18 on dist-test-slave-bwpp"
server_key: "42c597dfa97263a6fb534cca66815e9e"
server_key_iv: "26d45b054dafcb447361b195dfdad9b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:18.898079 5643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "297b60a9d72c400f926b8c145525bd0e"
format_stamp: "Formatted at 2026-06-20 12:25:18 on dist-test-slave-bwpp"
server_key: "42c597dfa97263a6fb534cca66815e9e"
server_key_iv: "26d45b054dafcb447361b195dfdad9b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:18.901780 5643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:25:18.904178 5657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:18.905368 5643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:18.905514 5643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "297b60a9d72c400f926b8c145525bd0e"
format_stamp: "Formatted at 2026-06-20 12:25:18 on dist-test-slave-bwpp"
server_key: "42c597dfa97263a6fb534cca66815e9e"
server_key_iv: "26d45b054dafcb447361b195dfdad9b8"
server_key_version: "encryptionkey@0"
I20260620 12:25:18.905628 5643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:18.916915 5643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:18.917565 5643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:18.917780 5643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:18.925819 5709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38683 every 8 connection(s)
I20260620 12:25:18.925817 5643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38683
I20260620 12:25:18.926890 5643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:18.927188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5643
I20260620 12:25:18.927279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:18.927646 392 external_mini_cluster.cc:1428] Setting key 68efbdf58358498cd17966e04cab74b4
I20260620 12:25:18.930261 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:18.936678 5710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e: Bootstrap starting.
I20260620 12:25:18.936717 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.928983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.0.98.62:38683)
0620 12:25:18.929606 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.929622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.929656 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:25:18.930284 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.930300 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.930806 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.931082 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:18.931429 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.932362 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.933549 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.933989 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.934101 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:18.935066 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:18.935071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:18.935082 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:18.935436 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:18.935488 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:18.935518 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:18.935590 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:18.935771 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 12:25:18.936052 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260620 12:25:18.939186 5710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:18.939532 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:25:18.940073 5710 log.cc:824] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:18.941984 5710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e: No bootstrap required, opened a new log
I20260620 12:25:18.945536 5710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297b60a9d72c400f926b8c145525bd0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 12:25:18.945746 5710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:18.945781 5710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297b60a9d72c400f926b8c145525bd0e, State: Initialized, Role: FOLLOWER
I20260620 12:25:18.946252 5710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297b60a9d72c400f926b8c145525bd0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 12:25:18.946381 5710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:18.946470 5710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:18.946581 5710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:18.947666 5710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297b60a9d72c400f926b8c145525bd0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 12:25:18.948037 5710 leader_election.cc:302] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 297b60a9d72c400f926b8c145525bd0e; no voters:
I20260620 12:25:18.948339 5710 leader_election.cc:288] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:18.948460 5715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:18.948695 5715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [term 1 LEADER]: Becoming Leader. State: Replica: 297b60a9d72c400f926b8c145525bd0e, State: Running, Role: LEADER
I20260620 12:25:18.949016 5715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297b60a9d72c400f926b8c145525bd0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 12:25:18.949523 5710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:18.950778 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "297b60a9d72c400f926b8c145525bd0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297b60a9d72c400f926b8c145525bd0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } } }
I20260620 12:25:18.950778 5717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 297b60a9d72c400f926b8c145525bd0e. Latest consensus state: current_term: 1 leader_uuid: "297b60a9d72c400f926b8c145525bd0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297b60a9d72c400f926b8c145525bd0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } } }
I20260620 12:25:18.950942 5717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:18.950942 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:18.951393 5725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:18.954412 5725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:18.959117 5725 catalog_manager.cc:1347] Generated new cluster ID: fe1a440b24b44b64b4ab539936369669
I20260620 12:25:18.959182 5725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:18.971491 5725 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:18.972491 5725 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:18.976924 5725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 297b60a9d72c400f926b8c145525bd0e: Generated new TSK 0
I20260620 12:25:18.977682 5725 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 12:25:19.002107 5740 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 12:25:19.002218 5740 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 12:25:19.002916 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:18.987063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.0.98.62:38683)
0620 12:25:18.987207 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:18.987212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:18.987374 (+ 162us) server_negotiation.cc:407] Connection header received
0620 12:25:18.987519 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:18.987522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:18.987576 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:18.987693 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:25:18.987913 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.988708 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.989558 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.989904 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.990014 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:18.990290 (+ 276us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:25:19.002468 (+ 12178us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 12:25:19.002766 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60764: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 12:25:19.002982 5713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60764: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 12:25:19.003051 5739 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:18.986943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38683 (local address 127.0.0.1:60764)
0620 12:25:18.987286 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:18.987298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:18.987401 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:18.987695 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:18.987697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:18.987703 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:25:18.987802 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:18.987807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.988844 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.988847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:18.989453 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:18.989460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:18.990019 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:18.990038 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:18.990104 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:18.990125 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:25:19.002713 (+ 12588us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 12:25:19.002863 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260620 12:25:19.003775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5643
2026-06-20T12:25:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (291 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 12:25:19.043411 392 mini_oidc.cc:152] Starting JWKS server
I20260620 12:25:19.044245 392 webserver.cc:460] Webserver started at https://127.0.0.1:40909/ using document root <none> and password file <none>
I20260620 12:25:19.044343 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:25:19.044569 392 webserver.cc:460] Webserver started at http://127.0.0.1:33137/ using document root <none> and password file <none>
2026-06-20T12:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:19Z Disabled control of system clock
I20260620 12:25:19.053610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37537
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37537
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40909/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 12:25:19.163586 5746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:19.163822 5746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:19.163859 5746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:19.168171 5746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:19.168231 5746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:19.168247 5746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:19.168264 5746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:19.168279 5746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:25:19.168303 5746 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:25:19.168336 5746 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40909/jwks.json
I20260620 12:25:19.173031 5746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40909/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:19.173959 5746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:19.175374 5746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:19.183512 5754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:19.183552 5751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:19.183542 5752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:19.183770 5746 server_base.cc:1034] running on GCE node
I20260620 12:25:19.184250 5746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:19.185261 5746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:19.186415 5746 hybrid_clock.cc:648] HybridClock initialized: now 1781958319186398 us; error 28 us; skew 500 ppm
I20260620 12:25:19.188228 5746 webserver.cc:460] Webserver started at http://127.0.98.62:34295/ using document root <none> and password file <none>
I20260620 12:25:19.188831 5746 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:19.188892 5746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:19.189054 5746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:19.191040 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa"
format_stamp: "Formatted at 2026-06-20 12:25:19 on dist-test-slave-bwpp"
server_key: "94205b533d713704645d1071fcb45939"
server_key_iv: "9e673a4f8709601ee61e6ef475d9eeae"
server_key_version: "encryptionkey@0"
I20260620 12:25:19.191540 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa"
format_stamp: "Formatted at 2026-06-20 12:25:19 on dist-test-slave-bwpp"
server_key: "94205b533d713704645d1071fcb45939"
server_key_iv: "9e673a4f8709601ee61e6ef475d9eeae"
server_key_version: "encryptionkey@0"
I20260620 12:25:19.195003 5746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:25:19.197240 5760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:19.198475 5746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:19.198626 5746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa"
format_stamp: "Formatted at 2026-06-20 12:25:19 on dist-test-slave-bwpp"
server_key: "94205b533d713704645d1071fcb45939"
server_key_iv: "9e673a4f8709601ee61e6ef475d9eeae"
server_key_version: "encryptionkey@0"
I20260620 12:25:19.198776 5746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:19.212426 5746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:19.213053 5746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:19.213235 5746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:19.220049 5746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37537
I20260620 12:25:19.220064 5812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37537 every 8 connection(s)
I20260620 12:25:19.220963 5746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:19.224040 5813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:19.229027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5746
I20260620 12:25:19.229125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:19.229415 392 external_mini_cluster.cc:1428] Setting key be0a7179175b1d2e4e773a5bd69e7313
I20260620 12:25:19.229542 5813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa: Bootstrap starting.
I20260620 12:25:19.232316 5813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:19.233165 5813 log.cc:824] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:19.235318 5813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa: No bootstrap required, opened a new log
I20260620 12:25:19.239892 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:19.230835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.0.98.62:37537)
0620 12:25:19.231481 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:19.231509 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:19.231550 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:25:19.232399 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:19.232423 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:19.232940 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:19.233309 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:19.233725 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.235152 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.236054 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.236740 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.236896 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:19.237714 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:19.237720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:19.237736 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:19.238235 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:19.238303 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:19.238329 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:19.238432 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:19.238568 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 12:25:19.238932 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260620 12:25:19.239859 5813 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37537 } }
I20260620 12:25:19.240156 5813 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:19.240232 5813 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd53a3c1edaa4b368ad37a4bc2ced5aa, State: Initialized, Role: FOLLOWER
I20260620 12:25:19.240671 5813 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37537 } }
I20260620 12:25:19.240792 5813 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:19.240875 5813 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:19.240970 5813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:19.241874 5813 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37537 } }
I20260620 12:25:19.242223 5813 leader_election.cc:302] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd53a3c1edaa4b368ad37a4bc2ced5aa; no voters:
I20260620 12:25:19.242558 5813 leader_election.cc:288] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:19.242645 5818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:19.242777 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:25:19.242868 5818 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [term 1 LEADER]: Becoming Leader. State: Replica: bd53a3c1edaa4b368ad37a4bc2ced5aa, State: Running, Role: LEADER
I20260620 12:25:19.243144 5818 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37537 } }
I20260620 12:25:19.243835 5813 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:19.244345 5820 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd53a3c1edaa4b368ad37a4bc2ced5aa. Latest consensus state: current_term: 1 leader_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37537 } } }
I20260620 12:25:19.244467 5820 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:19.244683 5819 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd53a3c1edaa4b368ad37a4bc2ced5aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37537 } } }
I20260620 12:25:19.244810 5819 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:19.245095 5823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:19.248441 5823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:19.253839 5823 catalog_manager.cc:1347] Generated new cluster ID: cda8ec8d247d459fa4834682c915f33c
I20260620 12:25:19.253922 5823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:19.271471 5823 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:19.272653 5823 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:19.281266 5823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd53a3c1edaa4b368ad37a4bc2ced5aa: Generated new TSK 0
I20260620 12:25:19.282022 5823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:19.295420 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:19.290588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.0.98.62:37537)
0620 12:25:19.290740 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:19.290745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:19.290877 (+ 132us) server_negotiation.cc:407] Connection header received
0620 12:25:19.291000 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:19.291003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:19.291058 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:19.291168 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:25:19.291364 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.292170 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.292961 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.293312 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.293411 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:19.293656 (+ 245us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:25:19.295051 (+ 1395us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40909/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 12:25:19.295270 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40909/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 12:25:19.295481 5816 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40909/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 12:25:19.297205 5842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:19.290501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37537 (local address 127.0.0.1:40406)
0620 12:25:19.290798 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:19.290810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:19.290900 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:19.291169 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:19.291171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:19.291177 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:25:19.291270 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:19.291274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.292267 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.292269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:19.292861 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:19.292867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.293403 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.293421 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:19.293482 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:19.293502 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:25:19.295207 (+ 1705us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40909/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 12:25:19.297041 (+ 1834us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40909/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1579,"thread_start_us":100,"threads_started":1}
I20260620 12:25:19.297876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5746
E20260620 12:25:19.306541 5843 webserver.cc:533] Webserver: sslize failed:
E20260620 12:25:19.306602 5843 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T12:25:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (292 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T12:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:19Z Disabled control of system clock
I20260620 12:25:19.344384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35727
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35727
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/testchainca.cert with env {}
W20260620 12:25:19.451284 5847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:19.451529 5847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:19.451576 5847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:19.455554 5847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:19.455605 5847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:19.455619 5847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:19.455636 5847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:19.455674 5847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:25:19.455705 5847 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:25:19.455731 5847 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 12:25:19.461001 5847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:19.462015 5847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:19.463068 5847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:19.471095 5852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:19.471184 5853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:19.471102 5855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:19.471302 5847 server_base.cc:1034] running on GCE node
I20260620 12:25:19.471908 5847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:19.472980 5847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:19.474148 5847 hybrid_clock.cc:648] HybridClock initialized: now 1781958319474119 us; error 40 us; skew 500 ppm
I20260620 12:25:19.476029 5847 webserver.cc:460] Webserver started at http://127.0.98.62:44593/ using document root <none> and password file <none>
I20260620 12:25:19.476601 5847 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:19.476660 5847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:19.476897 5847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:19.478942 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "6962bfe103b44147ac336d40ff80c6c4"
format_stamp: "Formatted at 2026-06-20 12:25:19 on dist-test-slave-bwpp"
server_key: "653584c040b3b61297f7fe5af64079d5"
server_key_iv: "9594638fe13a8fdf8f841fe4e4058e70"
server_key_version: "encryptionkey@0"
I20260620 12:25:19.479492 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "6962bfe103b44147ac336d40ff80c6c4"
format_stamp: "Formatted at 2026-06-20 12:25:19 on dist-test-slave-bwpp"
server_key: "653584c040b3b61297f7fe5af64079d5"
server_key_iv: "9594638fe13a8fdf8f841fe4e4058e70"
server_key_version: "encryptionkey@0"
I20260620 12:25:19.482932 5847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:25:19.485208 5861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:19.486377 5847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:19.486495 5847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "6962bfe103b44147ac336d40ff80c6c4"
format_stamp: "Formatted at 2026-06-20 12:25:19 on dist-test-slave-bwpp"
server_key: "653584c040b3b61297f7fe5af64079d5"
server_key_iv: "9594638fe13a8fdf8f841fe4e4058e70"
server_key_version: "encryptionkey@0"
I20260620 12:25:19.486596 5847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:19.510015 5847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:19.510764 5847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:19.510958 5847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:19.517939 5847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35727
I20260620 12:25:19.517949 5913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35727 every 8 connection(s)
I20260620 12:25:19.518966 5847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:19.519693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5847
I20260620 12:25:19.519819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:19.520136 392 external_mini_cluster.cc:1428] Setting key 4f1faeea6a999c38bdddd470dc6a53ff
I20260620 12:25:19.522300 5914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:19.527704 5914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4: Bootstrap starting.
I20260620 12:25:19.529157 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:19.521435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.0.98.62:35727)
0620 12:25:19.522068 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:19.522082 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:19.522116 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:25:19.522870 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:19.522887 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:19.523255 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:19.523598 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:19.524010 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.524968 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.525817 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.526273 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.526452 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:19.527437 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:19.527443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:19.527455 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:19.527897 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:19.527954 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:19.527974 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:19.528077 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:19.528233 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 12:25:19.528488 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
I20260620 12:25:19.529922 5914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:19.530634 5914 log.cc:824] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:19.531793 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:25:19.532665 5914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4: No bootstrap required, opened a new log
I20260620 12:25:19.536026 5914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962bfe103b44147ac336d40ff80c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260620 12:25:19.536221 5914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:19.536305 5914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6962bfe103b44147ac336d40ff80c6c4, State: Initialized, Role: FOLLOWER
I20260620 12:25:19.536765 5914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962bfe103b44147ac336d40ff80c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260620 12:25:19.536907 5914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:19.536978 5914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:19.537107 5914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:19.538022 5914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962bfe103b44147ac336d40ff80c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260620 12:25:19.538404 5914 leader_election.cc:302] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6962bfe103b44147ac336d40ff80c6c4; no voters:
I20260620 12:25:19.538786 5914 leader_election.cc:288] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:19.538885 5919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:19.539098 5919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [term 1 LEADER]: Becoming Leader. State: Replica: 6962bfe103b44147ac336d40ff80c6c4, State: Running, Role: LEADER
I20260620 12:25:19.539382 5919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962bfe103b44147ac336d40ff80c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260620 12:25:19.539909 5914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:19.540961 5920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6962bfe103b44147ac336d40ff80c6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962bfe103b44147ac336d40ff80c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } } }
I20260620 12:25:19.541095 5920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:19.541363 5921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6962bfe103b44147ac336d40ff80c6c4. Latest consensus state: current_term: 1 leader_uuid: "6962bfe103b44147ac336d40ff80c6c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6962bfe103b44147ac336d40ff80c6c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } } }
I20260620 12:25:19.541460 5921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:19.541558 5928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:19.544230 5928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:19.549343 5928 catalog_manager.cc:1347] Generated new cluster ID: 5f1ed791193445169840080e2b3234bb
I20260620 12:25:19.549412 5928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:19.564678 5928 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:19.565672 5928 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:19.569805 5928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6962bfe103b44147ac336d40ff80c6c4: Generated new TSK 0
I20260620 12:25:19.570431 5928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:19.662256 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:19.583923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.0.98.62:35727)
0620 12:25:19.584068 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:19.584076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:19.584232 (+ 156us) server_negotiation.cc:407] Connection header received
0620 12:25:19.584385 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:19.584387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:19.584430 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:19.584518 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:25:19.584770 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.585278 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.586486 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.586741 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.586817 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:19.587120 (+ 303us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:25:19.661875 (+ 74755us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 12:25:19.662136 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53302: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 12:25:19.662360 5917 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53302: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 12:25:19.662529 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:19.583840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35727 (local address 127.0.0.1:53302)
0620 12:25:19.584141 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:19.584156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:19.584269 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:19.584541 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:19.584545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:19.584553 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:25:19.584671 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:19.584677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.585388 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.585391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:19.586376 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:19.586387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:19.586838 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:19.586861 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:19.586950 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:19.586976 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:25:19.662120 (+ 75144us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 12:25:19.662343 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
I20260620 12:25:19.663306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5847
2026-06-20T12:25:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (346 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:19 dist-test-slave-bwpp krb5kdc[5955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:19 dist-test-slave-bwpp krb5kdc[5955](info): set up 2 sockets
Jun 20 12:25:19 dist-test-slave-bwpp krb5kdc[5955](info): commencing operation
krb5kdc: starting...
W20260620 12:25:20.935532 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:21 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260620 12:25:21.930560 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.896s user 0.000s sys 0.003s
I20260620 12:25:21.931136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41039
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:22.044287 5968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:22.044607 5968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:22.044682 5968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:22.048874 5968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:22.048969 5968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:22.049007 5968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:22.049062 5968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:22.049103 5968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:22.054487 5968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:22.055619 5968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:22.056649 5968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:22.065124 5974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:22.065128 5976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:22.065135 5973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:22.065335 5968 server_base.cc:1034] running on GCE node
I20260620 12:25:22.065938 5968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:22.067061 5968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:22.068254 5968 hybrid_clock.cc:648] HybridClock initialized: now 1781958322068222 us; error 44 us; skew 500 ppm
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.075729 5968 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:22.076705 5968 webserver.cc:460] Webserver started at http://127.0.98.62:41803/ using document root <none> and password file <none>
I20260620 12:25:22.077296 5968 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:22.077430 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:22.077682 5968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:22.079830 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "125f5b743dfb47ee8d5765b66bf68783"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "0d90302cb3f625fd6ca3c53baf90dae4"
server_key_iv: "b9b55373e4950a8fb26a88c70640ee94"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.080403 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "125f5b743dfb47ee8d5765b66bf68783"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "0d90302cb3f625fd6ca3c53baf90dae4"
server_key_iv: "b9b55373e4950a8fb26a88c70640ee94"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.084203 5968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:25:22.086805 5983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.088121 5968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:22.088277 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "125f5b743dfb47ee8d5765b66bf68783"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "0d90302cb3f625fd6ca3c53baf90dae4"
server_key_iv: "b9b55373e4950a8fb26a88c70640ee94"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.088425 5968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:22.110188 5968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:22.113583 5968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:22.113835 5968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:22.122095 5968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41039
I20260620 12:25:22.122123 6035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41039 every 8 connection(s)
I20260620 12:25:22.123095 5968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:22.125771 6036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:22.128273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 5968
I20260620 12:25:22.128383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:22.128685 392 external_mini_cluster.cc:1428] Setting key 27ba1a0699dc0fd74689ef1185baf0ce
I20260620 12:25:22.131937 6036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783: Bootstrap starting.
I20260620 12:25:22.134480 6036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:22.135351 6036 log.cc:824] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:22.137436 6036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783: No bootstrap required, opened a new log
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:22.141790 6036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125f5b743dfb47ee8d5765b66bf68783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41039 } }
I20260620 12:25:22.142076 6036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:22.142150 6036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125f5b743dfb47ee8d5765b66bf68783, State: Initialized, Role: FOLLOWER
I20260620 12:25:22.142686 6036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125f5b743dfb47ee8d5765b66bf68783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41039 } }
I20260620 12:25:22.142853 6036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:22.142975 6036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:22.143107 6036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:22.144341 6036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125f5b743dfb47ee8d5765b66bf68783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41039 } }
I20260620 12:25:22.144716 6036 leader_election.cc:302] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125f5b743dfb47ee8d5765b66bf68783; no voters:
I20260620 12:25:22.145078 6036 leader_election.cc:288] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:22.145159 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.130159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.98.62:41039)
0620 12:25:22.130579 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.130595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.130628 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:25:22.131303 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.131319 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.131653 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.131938 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.132846 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.133827 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.134992 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.135600 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.135759 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.138570 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:22.138576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:22.138592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:22.138622 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:22.141204 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.141652 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.141657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.141662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.141814 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.142235 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.142239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.142241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.142800 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:22.143646 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:22.143806 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 12:25:22.144126 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":147,"threads_started":1}
I20260620 12:25:22.145247 6041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:22.145874 6041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [term 1 LEADER]: Becoming Leader. State: Replica: 125f5b743dfb47ee8d5765b66bf68783, State: Running, Role: LEADER
I20260620 12:25:22.146443 6036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:22.146479 6041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125f5b743dfb47ee8d5765b66bf68783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41039 } }
I20260620 12:25:22.148389 6042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "125f5b743dfb47ee8d5765b66bf68783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125f5b743dfb47ee8d5765b66bf68783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41039 } } }
I20260620 12:25:22.148533 6042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:22.148823 6043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125f5b743dfb47ee8d5765b66bf68783. Latest consensus state: current_term: 1 leader_uuid: "125f5b743dfb47ee8d5765b66bf68783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125f5b743dfb47ee8d5765b66bf68783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41039 } } }
I20260620 12:25:22.149159 6043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783 [sys.catalog]: This master's current role is: LEADER
W20260620 12:25:22.151779 6055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:25:22.151867 6055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:25:22.151969 6056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:22.152710 6056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:22.158252 6056 catalog_manager.cc:1347] Generated new cluster ID: c142f3f4a82d47a99354813709ba7a69
I20260620 12:25:22.158337 6056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:22.175606 6056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:22.176770 6056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:22.189093 6056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125f5b743dfb47ee8d5765b66bf68783: Generated new TSK 0
I20260620 12:25:22.189785 6056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:25:22.219797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41039
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:22.351660 6062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:22.351950 6062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:22.352002 6062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:22.356290 6062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:22.356389 6062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:22.356490 6062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:22.364305 6062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:22.365614 6062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:22.366885 6062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:22.377864 6067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:22.377843 6070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:22.378165 6062 server_base.cc:1034] running on GCE node
W20260620 12:25:22.377844 6068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:22.378577 6062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:22.379266 6062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:22.380505 6062 hybrid_clock.cc:648] HybridClock initialized: now 1781958322380487 us; error 30 us; skew 500 ppm
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.383399 6062 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:22.384254 6062 webserver.cc:460] Webserver started at http://127.0.98.1:35537/ using document root <none> and password file <none>
I20260620 12:25:22.384824 6062 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:22.384913 6062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:22.385140 6062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:22.387344 6062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5227589123142efac145f875dde76fa"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "4d84cccdd880dfaff86404d21e6c324a"
server_key_iv: "8761f6f2b5bb29e7f85c97d4d1ac4198"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.387915 6062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5227589123142efac145f875dde76fa"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "4d84cccdd880dfaff86404d21e6c324a"
server_key_iv: "8761f6f2b5bb29e7f85c97d4d1ac4198"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.391870 6062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:25:22.394295 6077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.395565 6062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:22.395695 6062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "d5227589123142efac145f875dde76fa"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "4d84cccdd880dfaff86404d21e6c324a"
server_key_iv: "8761f6f2b5bb29e7f85c97d4d1ac4198"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.395812 6062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:22.407136 6062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:22.409832 6062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:22.410065 6062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:22.410753 6062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:22.411552 6062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:22.411597 6062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.411638 6062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:22.411659 6062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.421178 6062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39109
I20260620 12:25:22.421200 6190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39109 every 8 connection(s)
I20260620 12:25:22.422170 6062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:22.427073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6062
I20260620 12:25:22.427175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:22.427479 392 external_mini_cluster.cc:1428] Setting key 67aee6e7f2aaf585d24e2ef834461860
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:25:22.438241 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.424114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38319 (local address 127.0.98.62:41039)
0620 12:25:22.424269 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.424273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.425126 (+ 853us) server_negotiation.cc:407] Connection header received
0620 12:25:22.426562 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.426566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.426626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.426772 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.428280 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.428884 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.430777 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.431213 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.431334 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.434158 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:22.434163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:22.434169 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:22.434206 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:22.436043 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.436568 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.436574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.436576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.436675 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.437009 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.437014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.437015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.437299 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:22.437413 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:22.437973 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 12:25:22.438114 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:25:22.439215 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.424503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41039 (local address 127.0.98.1:38319)
0620 12:25:22.424969 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:22.425017 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:22.426320 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:22.426970 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:22.426978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:22.427454 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:22.428127 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:22.428145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.429024 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.429027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:22.430571 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:22.430586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.431374 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.431377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:22.431528 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.432195 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:22.432212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:22.433966 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:22.436166 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:22.436171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:22.436185 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:22.436463 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:22.436770 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:22.436772 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:22.436774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:22.436904 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:22.437413 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:22.437418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:22.437665 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 12:25:22.437892 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 12:25:22.438178 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:22.440704 6191 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41039
I20260620 12:25:22.440971 6191 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:22.441488 6191 heartbeater.cc:511] Master 127.0.98.62:41039 requested a full tablet report, sending...
I20260620 12:25:22.443151 6000 ts_manager.cc:194] Registered new tserver with Master: d5227589123142efac145f875dde76fa (127.0.98.1:39109)
I20260620 12:25:22.444293 6000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:22.457785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41039
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:22.569006 6196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:22.569360 6196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:22.569444 6196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:22.573844 6196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:22.573951 6196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:22.574050 6196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:22.579582 6196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:22.580737 6196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:22.581853 6196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:22.592057 6204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:22.592046 6201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:22.592290 6196 server_base.cc:1034] running on GCE node
W20260620 12:25:22.592065 6202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:22.592753 6196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:22.593403 6196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:22.594587 6196 hybrid_clock.cc:648] HybridClock initialized: now 1781958322594553 us; error 49 us; skew 500 ppm
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.597299 6196 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:22.598287 6196 webserver.cc:460] Webserver started at http://127.0.98.2:40469/ using document root <none> and password file <none>
I20260620 12:25:22.598960 6196 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:22.599030 6196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:22.599206 6196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:22.601346 6196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "6dd7aeb6f8c94be9904031320c9975e8"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "ddaa44e85924f37c083630187cdf4bda"
server_key_iv: "8d89a2ed9fec27459ee5cf3d6ce35b6e"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.601869 6196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6dd7aeb6f8c94be9904031320c9975e8"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "ddaa44e85924f37c083630187cdf4bda"
server_key_iv: "8d89a2ed9fec27459ee5cf3d6ce35b6e"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.605607 6196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 12:25:22.608100 6211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.609305 6196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:22.609403 6196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "6dd7aeb6f8c94be9904031320c9975e8"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "ddaa44e85924f37c083630187cdf4bda"
server_key_iv: "8d89a2ed9fec27459ee5cf3d6ce35b6e"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.609500 6196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:22.626304 6196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:22.629388 6196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:22.629554 6196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:22.630112 6196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:22.630975 6196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:22.631026 6196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.631065 6196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:22.631086 6196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.640411 6196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42957
I20260620 12:25:22.640432 6324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42957 every 8 connection(s)
I20260620 12:25:22.641351 6196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:22.644047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6196
I20260620 12:25:22.644160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:22.644475 392 external_mini_cluster.cc:1428] Setting key f7806ec2730ed956221c1a3256f561f0
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:25:22.656626 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.643259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57117 (local address 127.0.98.62:41039)
0620 12:25:22.643403 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.643407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.644524 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 12:25:22.646293 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.646296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.646341 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.646411 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.648069 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.648814 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.650022 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.650265 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.650382 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.652988 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:22.652992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:22.652994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:22.653029 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:22.654484 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.655041 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.655045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.655047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.655151 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.655537 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.655540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.655541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.655714 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:22.655787 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:22.656340 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 12:25:22.656440 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:25:22.657545 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.643739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41039 (local address 127.0.98.2:57117)
0620 12:25:22.644335 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:22.644395 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:22.646039 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:22.646644 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:22.646674 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:22.647229 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:22.647925 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:22.647942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.648961 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.648967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:22.649880 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:22.649889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.650419 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.650423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:22.650595 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.651340 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:22.651357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:22.652815 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:22.654605 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:22.654611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:22.654627 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:22.654918 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:22.655251 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:22.655253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:22.655255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:22.655435 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:22.655810 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:22.655816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:22.656113 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 12:25:22.656267 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 12:25:22.656654 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":142,"threads_started":1}
I20260620 12:25:22.658839 6325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41039
I20260620 12:25:22.659096 6325 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:22.659586 6325 heartbeater.cc:511] Master 127.0.98.62:41039 requested a full tablet report, sending...
I20260620 12:25:22.660703 6000 ts_manager.cc:194] Registered new tserver with Master: 6dd7aeb6f8c94be9904031320c9975e8 (127.0.98.2:42957)
I20260620 12:25:22.661588 6000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:22.671873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41039
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:22.783362 6330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:22.783669 6330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:22.783751 6330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:22.788107 6330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:22.788198 6330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:22.788290 6330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:22.793803 6330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:22.795059 6330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:22.796129 6330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:22.806252 6336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:22.806262 6335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:22.806326 6330 server_base.cc:1034] running on GCE node
W20260620 12:25:22.806262 6338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:22.807000 6330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:22.807642 6330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:22.808801 6330 hybrid_clock.cc:648] HybridClock initialized: now 1781958322808789 us; error 33 us; skew 500 ppm
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.811645 6330 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:22.812595 6330 webserver.cc:460] Webserver started at http://127.0.98.3:34913/ using document root <none> and password file <none>
I20260620 12:25:22.813127 6330 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:22.813185 6330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:22.813349 6330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:22.815569 6330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "6909e2e4b62f40e6801932e5d4874f6d"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "acefeda0d650cff7abf16162c90a29e0"
server_key_iv: "b2275da65c09b58795e4638ee08bd760"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.816088 6330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6909e2e4b62f40e6801932e5d4874f6d"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "acefeda0d650cff7abf16162c90a29e0"
server_key_iv: "b2275da65c09b58795e4638ee08bd760"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.819748 6330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:25:22.822120 6345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.823457 6330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:22.823585 6330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "6909e2e4b62f40e6801932e5d4874f6d"
format_stamp: "Formatted at 2026-06-20 12:25:22 on dist-test-slave-bwpp"
server_key: "acefeda0d650cff7abf16162c90a29e0"
server_key_iv: "b2275da65c09b58795e4638ee08bd760"
server_key_version: "encryptionkey@0"
I20260620 12:25:22.823690 6330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:22.845086 6330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:22.848567 6330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:22.848770 6330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:22.849377 6330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:22.850477 6330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:22.850567 6330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:22.850636 6330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:22.850714 6330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 12:25:22.859611 6330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46639
I20260620 12:25:22.859632 6458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46639 every 8 connection(s)
I20260620 12:25:22.860539 6330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:22.868440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6330
I20260620 12:25:22.868579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:22.868873 392 external_mini_cluster.cc:1428] Setting key 86c5c78afc7ae5dd81db4b48e32003ca
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:22.875029 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.862502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54689 (local address 127.0.98.62:41039)
0620 12:25:22.862638 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.862641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.863541 (+ 900us) server_negotiation.cc:407] Connection header received
0620 12:25:22.864972 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.864975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.865035 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.865144 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.866382 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.866999 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.868158 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.868482 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.868618 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.871527 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:22.871530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:22.871533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:22.871563 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:22.872844 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.873386 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.873391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.873393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.873473 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.873897 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.873902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.873904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.874095 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:22.874200 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:22.874743 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 12:25:22.874863 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:22.875897 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.862936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41039 (local address 127.0.98.3:54689)
0620 12:25:22.863387 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:22.863431 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:22.864742 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:22.865342 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:22.865350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:22.865732 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:22.866229 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:22.866244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.867123 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.867126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:22.868022 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:22.868031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.868687 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.868691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:22.868852 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.869537 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:22.869556 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:22.871361 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:22.872986 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:22.873000 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:22.873016 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:22.873277 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:22.873590 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:22.873593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:22.873595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:22.873799 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:22.874210 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:22.874218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:22.874478 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 12:25:22.874696 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 12:25:22.875055 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260620 12:25:22.877007 6459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41039
I20260620 12:25:22.877267 6459 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:22.877797 6459 heartbeater.cc:511] Master 127.0.98.62:41039 requested a full tablet report, sending...
I20260620 12:25:22.878844 6000 ts_manager.cc:194] Registered new tserver with Master: 6909e2e4b62f40e6801932e5d4874f6d (127.0.98.3:46639)
I20260620 12:25:22.879412 6000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54689
I20260620 12:25:22.882889 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/test-admin.keytab.
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.899890 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 12:25:22.902930 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5cc' of type FILE
W20260620 12:25:22.902971 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:22.901427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.98.1:39109)
0620 12:25:22.901721 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.901757 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.901785 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:25:22.901844 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.901850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.902055 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.902194 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.902724 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57844: BlockingRecv error: recv got EOF from 127.0.0.1:57844 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.904138 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.905277 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:25:22.915234 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.906212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.98.1:39109)
0620 12:25:22.906385 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.906389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.906440 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:25:22.906567 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.906570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.906612 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.906787 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.907302 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.908154 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.909010 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.909439 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.909543 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.911946 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:22.911955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:22.911969 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:22.911991 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:22.913515 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.914027 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.914032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.914034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.914106 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.914335 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.914337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.914338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.914616 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:22.914827 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:22.914975 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 12:25:22.915095 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 12:25:22.918835 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:22.918005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.0.98.1:39109)
0620 12:25:22.918105 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.918108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.918175 (+ 67us) server_negotiation.cc:407] Connection header received
0620 12:25:22.918247 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.918249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.918281 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.918349 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.918747 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57852: BlockingRecv error: recv got EOF from 127.0.0.1:57852 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 12:25:22.918928 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.919924 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.920745 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:25:22.929355 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.921545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.0.98.1:39109)
0620 12:25:22.921664 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.921669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.921773 (+ 104us) server_negotiation.cc:407] Connection header received
0620 12:25:22.921909 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.921912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.921960 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.922046 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.922517 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.923093 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.923890 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.924122 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.924243 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.926433 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:22.926437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:22.926438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:22.926460 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:22.927905 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.928342 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.928347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.928352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.928429 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.928655 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.928658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.928659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.928808 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:22.928901 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:22.929150 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 12:25:22.929239 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 12:25:22.932225 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:22.931361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.0.98.1:39109)
0620 12:25:22.931476 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.931479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.931523 (+ 44us) server_negotiation.cc:407] Connection header received
0620 12:25:22.931643 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.931645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.931677 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.931738 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.932137 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57868: BlockingRecv error: recv got EOF from 127.0.0.1:57868 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 12:25:22.932461 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781958290914605-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.933588 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:22.934583 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:25:22 dist-test-slave-bwpp krb5kdc[5955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:25:22.943929 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:22.935490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.98.1:39109)
0620 12:25:22.935597 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:22.935600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:22.935628 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:25:22.935716 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:22.935718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:22.935752 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:22.935829 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:22.936266 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.936833 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.937822 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:22.938189 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:22.938291 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:22.941031 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:22.941034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:22.941036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:22.941058 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:22.942510 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.942941 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.942944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.942945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.943015 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:22.943237 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:22.943239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:22.943240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:22.943389 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:22.943484 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:22.943717 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 12:25:22.943804 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:25:22.944967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6062
I20260620 12:25:22.950879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6196
I20260620 12:25:22.956213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6330
I20260620 12:25:22.961447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 5968
2026-06-20T12:25:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3061 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:23 dist-test-slave-bwpp krb5kdc[6479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:23 dist-test-slave-bwpp krb5kdc[6479](info): set up 2 sockets
Jun 20 12:25:23 dist-test-slave-bwpp krb5kdc[6479](info): commencing operation
krb5kdc: starting...
W20260620 12:25:24.007171 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 12:25:24.129740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:24.246239 6492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:24.246559 6492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:24.246670 6492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:24.250700 6492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:24.250806 6492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:24.250851 6492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:24.250910 6492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:24.250953 6492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:24.256372 6492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:24.257496 6492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:24.258554 6492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:24.266952 6500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:24.267037 6497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:24.267026 6498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:24.267174 6492 server_base.cc:1034] running on GCE node
I20260620 12:25:24.267835 6492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:24.268946 6492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:24.270143 6492 hybrid_clock.cc:648] HybridClock initialized: now 1781958324270115 us; error 52 us; skew 500 ppm
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:24.272923 6492 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 12:25:24.273797 6492 webserver.cc:460] Webserver started at http://127.0.98.62:38219/ using document root <none> and password file <none>
I20260620 12:25:24.274322 6492 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:24.274386 6492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:24.274550 6492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:24.276604 6492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "0c1403069df443ce8ec099e16da75a42"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "1753376c7edaa1c2089b2666573544e6"
server_key_iv: "6c6fd1e5188dc016b37038a67138fdc5"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.277137 6492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c1403069df443ce8ec099e16da75a42"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "1753376c7edaa1c2089b2666573544e6"
server_key_iv: "6c6fd1e5188dc016b37038a67138fdc5"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.280869 6492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 12:25:24.283293 6507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:24.284626 6492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:24.284737 6492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "0c1403069df443ce8ec099e16da75a42"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "1753376c7edaa1c2089b2666573544e6"
server_key_iv: "6c6fd1e5188dc016b37038a67138fdc5"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.284842 6492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:24.300379 6492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:24.303325 6492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:24.303485 6492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:24.310496 6492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45973
I20260620 12:25:24.310825 6559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45973 every 8 connection(s)
I20260620 12:25:24.311530 6492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:24.314421 6560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:24.315757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6492
I20260620 12:25:24.315901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:24.316229 392 external_mini_cluster.cc:1428] Setting key 3d791d4654f08be822b10c4c7d1f6ecc
I20260620 12:25:24.321002 6560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: Bootstrap starting.
I20260620 12:25:24.323446 6560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:24.324191 6560 log.cc:824] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:24.326401 6560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: No bootstrap required, opened a new log
I20260620 12:25:24.330039 6560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260620 12:25:24.330325 6560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:24.330413 6560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1403069df443ce8ec099e16da75a42, State: Initialized, Role: FOLLOWER
I20260620 12:25:24.330926 6560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260620 12:25:24.331035 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:24.317706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.98.62:45973)
0620 12:25:24.318160 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:24.318175 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:24.318212 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:25:24.318922 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:24.318941 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:24.319288 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:24.319650 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:24.320469 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.321555 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.322416 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.323001 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.323180 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:24.326113 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:24.326118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:24.326133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:24.326160 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:24.327946 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:24.328415 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:24.328421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:24.328425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:24.328538 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:24.328834 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:24.328838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:24.328839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:24.329238 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:24.329455 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:24.329778 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 12:25:24.330115 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260620 12:25:24.331099 6560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:24.331172 6560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:24.331293 6560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:24.332450 6560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260620 12:25:24.332800 6560 leader_election.cc:302] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1403069df443ce8ec099e16da75a42; no voters:
I20260620 12:25:24.333134 6560 leader_election.cc:288] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:24.333243 6565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:24.333446 6565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1403069df443ce8ec099e16da75a42, State: Running, Role: LEADER
I20260620 12:25:24.333778 6565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } }
I20260620 12:25:24.334381 6560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:24.335039 6567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c1403069df443ce8ec099e16da75a42. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } }
I20260620 12:25:24.335273 6567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:24.335489 6566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } }
I20260620 12:25:24.335583 6566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:24.335920 6574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:24.339275 6574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:24.345206 6574 catalog_manager.cc:1347] Generated new cluster ID: 4d6161b194794dc484afd4156351d26c
I20260620 12:25:24.345284 6574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:24.358037 6574 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:24.359222 6574 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:24.365478 6574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: Generated new TSK 0
I20260620 12:25:24.366482 6574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 12:25:24.397634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:24.511922 6586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:24.512199 6586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:24.512243 6586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:24.516355 6586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:24.516419 6586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:24.516479 6586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:24.521687 6586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:24.522785 6586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:24.523856 6586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:24.533777 6591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:24.533772 6594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:24.533772 6592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:24.534159 6586 server_base.cc:1034] running on GCE node
I20260620 12:25:24.534569 6586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:24.535282 6586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:24.536448 6586 hybrid_clock.cc:648] HybridClock initialized: now 1781958324536424 us; error 35 us; skew 500 ppm
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:24.539274 6586 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:25:24.540131 6586 webserver.cc:460] Webserver started at http://127.0.98.1:36737/ using document root <none> and password file <none>
I20260620 12:25:24.540714 6586 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:24.540807 6586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:24.541029 6586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:24.543108 6586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "f245e2fae6a64ed49ca6086fa18cb731"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "15ef800908e370a82e7eccdaf7bde3c2"
server_key_iv: "0f1a7cc95290231bb45e936cb783883b"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.543673 6586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f245e2fae6a64ed49ca6086fa18cb731"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "15ef800908e370a82e7eccdaf7bde3c2"
server_key_iv: "0f1a7cc95290231bb45e936cb783883b"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.547469 6586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:25:24.549834 6601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:24.551116 6586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:24.551250 6586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "f245e2fae6a64ed49ca6086fa18cb731"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "15ef800908e370a82e7eccdaf7bde3c2"
server_key_iv: "0f1a7cc95290231bb45e936cb783883b"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.551390 6586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:24.564646 6586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:24.567948 6586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:24.568148 6586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:24.568727 6586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:24.569581 6586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:24.569653 6586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:24.569722 6586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:24.569772 6586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:24.578475 6586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39003
I20260620 12:25:24.578496 6714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39003 every 8 connection(s)
I20260620 12:25:24.579388 6586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:24.583896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6586
I20260620 12:25:24.584034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:24.584326 392 external_mini_cluster.cc:1428] Setting key 3fc5aa2322c95a820454e6f0dd97c9e8
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:24.594480 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:24.581196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53077 (local address 127.0.98.62:45973)
0620 12:25:24.581326 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:24.581330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:24.582249 (+ 919us) server_negotiation.cc:407] Connection header received
0620 12:25:24.583660 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:24.583663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:24.583723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:24.583825 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:24.585144 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.585771 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.587612 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.588027 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.588142 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:24.590694 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:24.590701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:24.590708 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:24.590744 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:24.592443 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:24.592933 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:24.592940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:24.592942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:24.593041 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:24.593361 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:24.593364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:24.593366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:24.593598 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:24.593711 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:24.594175 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 12:25:24.594305 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:25:24.595234 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:24.581645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:53077)
0620 12:25:24.582105 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:24.582146 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:24.583445 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:24.584044 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:24.584051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:24.584499 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:24.585000 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:24.585021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.585909 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.585913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:24.587460 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:24.587475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.588163 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.588165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:24.588306 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:24.588904 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:24.588920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:24.590483 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:24.592561 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:24.592566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:24.592578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:24.592829 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:24.593138 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:24.593140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:24.593142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:24.593264 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:24.593701 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:24.593705 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:24.593927 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 12:25:24.594082 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 12:25:24.594323 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260620 12:25:24.596764 6715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260620 12:25:24.597013 6715 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:24.597575 6715 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260620 12:25:24.599170 6524 ts_manager.cc:194] Registered new tserver with Master: f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1:39003)
I20260620 12:25:24.600304 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 12:25:24.612089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:24.725461 6720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:24.725735 6720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:24.725816 6720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:24.730245 6720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:24.730304 6720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:24.730392 6720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:24.735929 6720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:24.737049 6720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:24.738150 6720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:24.747746 6725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:24.747772 6728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:24.747773 6726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:24.748111 6720 server_base.cc:1034] running on GCE node
I20260620 12:25:24.748517 6720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:24.749073 6720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:24.750233 6720 hybrid_clock.cc:648] HybridClock initialized: now 1781958324750215 us; error 30 us; skew 500 ppm
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:24.752884 6720 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:25:24.753700 6720 webserver.cc:460] Webserver started at http://127.0.98.2:44375/ using document root <none> and password file <none>
I20260620 12:25:24.754235 6720 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:24.754324 6720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:24.754582 6720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:24.756740 6720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "fe7a58f517d04d10aa1f256aa931244e"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3056e837a0939f82486b5c71dd75d624"
server_key_iv: "dbc2628541cf6a17c9589e064c5c5a88"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.757320 6720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fe7a58f517d04d10aa1f256aa931244e"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3056e837a0939f82486b5c71dd75d624"
server_key_iv: "dbc2628541cf6a17c9589e064c5c5a88"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.761000 6720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 12:25:24.763397 6735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:24.764647 6720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:25:24.764781 6720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "fe7a58f517d04d10aa1f256aa931244e"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3056e837a0939f82486b5c71dd75d624"
server_key_iv: "dbc2628541cf6a17c9589e064c5c5a88"
server_key_version: "encryptionkey@0"
I20260620 12:25:24.764892 6720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:24.796936 6720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:24.800457 6720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:24.800684 6720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:24.801316 6720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:24.802196 6720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:24.802270 6720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:24.802343 6720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:24.802387 6720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:24.811782 6720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39413
I20260620 12:25:24.811801 6848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39413 every 8 connection(s)
I20260620 12:25:24.812717 6720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:24.818805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6720
I20260620 12:25:24.818940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:24.819263 392 external_mini_cluster.cc:1428] Setting key 1a7cc21d8ab9b5a86241765bf75ffc0e
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:24.827461 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:24.814698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35253 (local address 127.0.98.62:45973)
0620 12:25:24.814857 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:24.814873 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:24.815714 (+ 841us) server_negotiation.cc:407] Connection header received
0620 12:25:24.817005 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:24.817008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:24.817057 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:24.817136 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:24.818263 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.818809 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.820340 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.820651 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.820766 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:24.823283 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:24.823297 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:24.823301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:24.823331 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:24.825008 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:24.825619 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:24.825623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:24.825624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:24.825701 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:24.826123 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:24.826125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:24.826127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:24.826379 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:24.826456 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:24.827197 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 12:25:24.827336 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:25:24.828692 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:24.815096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:35253)
0620 12:25:24.815549 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:24.815594 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:24.816809 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:24.817297 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:24.817307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:24.817655 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:24.818143 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:24.818156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.818980 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.818984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:24.820220 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:24.820231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:24.820805 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:24.820809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:24.820988 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:24.821622 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:24.821638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:24.823113 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:24.825158 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:24.825165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:24.825182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:24.825456 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:24.825801 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:24.825807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:24.825812 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:24.826017 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:24.826489 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:24.826506 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:24.826918 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 12:25:24.827150 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 12:25:24.827556 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:25:24.829905 6849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260620 12:25:24.830215 6849 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:24.830956 6849 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260620 12:25:24.832093 6524 ts_manager.cc:194] Registered new tserver with Master: fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413)
I20260620 12:25:24.832674 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 12:25:24.849229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:24.972539 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:24.972851 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:24.972931 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:24.977250 6854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:24.977372 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:24.977490 6854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:24.983261 6854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:24.984484 6854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:24.985654 6854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:24.995930 6862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:24.995968 6860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:24.996177 6854 server_base.cc:1034] running on GCE node
W20260620 12:25:24.995981 6859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:24.996754 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:24.997474 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:24.998703 6854 hybrid_clock.cc:648] HybridClock initialized: now 1781958324998668 us; error 44 us; skew 500 ppm
Jun 20 12:25:24 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:25.001575 6854 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:25:25.002477 6854 webserver.cc:460] Webserver started at http://127.0.98.3:35613/ using document root <none> and password file <none>
I20260620 12:25:25.003118 6854 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.003206 6854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.003441 6854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:25.005486 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "a440174e3fc34439b319063e963f916a"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3d9b3ff476571786b1f1908f12310f98"
server_key_iv: "530482b373b2417ba3e4440c2b8b1953"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.006080 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a440174e3fc34439b319063e963f916a"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3d9b3ff476571786b1f1908f12310f98"
server_key_iv: "530482b373b2417ba3e4440c2b8b1953"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.009943 6854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:25:25.012461 6869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.013680 6854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:25.013818 6854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "a440174e3fc34439b319063e963f916a"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3d9b3ff476571786b1f1908f12310f98"
server_key_iv: "530482b373b2417ba3e4440c2b8b1953"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.013934 6854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.041285 6854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:25.044862 6854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:25.045080 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:25.045725 6854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:25.046603 6854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:25.046700 6854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.046792 6854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:25.046847 6854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.055534 6854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46617
I20260620 12:25:25.055554 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46617 every 8 connection(s)
I20260620 12:25:25.056572 6854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:25.056766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6854
I20260620 12:25:25.056905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:25.057260 392 external_mini_cluster.cc:1428] Setting key 17b115de5c7d3dac9bdbbaa5381b25b2
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:25.071121 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.058525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42355 (local address 127.0.98.62:45973)
0620 12:25:25.058685 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.058690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.059791 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 12:25:25.061101 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.061104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.061149 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.061217 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.062374 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.063012 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.064303 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.064525 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.064628 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.067316 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.067328 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.067331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.067358 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.068805 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.069350 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.069353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.069354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.069435 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.069744 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.069747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.069748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.070013 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.070117 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.070797 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.070945 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:25:25.072127 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.059124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:42355)
0620 12:25:25.059638 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.059696 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.060898 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.061374 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.061381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.061744 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.062241 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.062255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.063117 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.063120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.064130 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.064159 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.064676 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.064679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.064829 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.065487 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.065503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.067090 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.068953 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.068962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.068987 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.069254 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.069526 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.069528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.069529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.069665 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.070149 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.070156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.070479 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.070759 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.071094 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260620 12:25:25.073215 6983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260620 12:25:25.073465 6983 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:25.073967 6983 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260620 12:25:25.075021 6524 ts_manager.cc:194] Registered new tserver with Master: a440174e3fc34439b319063e963f916a (127.0.98.3:46617)
I20260620 12:25:25.075552 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42355
I20260620 12:25:25.083688 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 12:25:25.110175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33011
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45973,127.0.98.61:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:25.223167 6988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:25.223497 6988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:25.223590 6988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:25.227723 6988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:25.227818 6988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:25.227855 6988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:25.227910 6988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:25.227947 6988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:25.233547 6988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45973,127.0.98.61:33011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
W20260620 12:25:25.233860 6988 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45973,127.0.98.61:33011'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 12:25:25.234977 6988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:25.236155 6988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:25.245052 6993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.245045 6996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.245045 6994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:25.245355 6988 server_base.cc:1034] running on GCE node
I20260620 12:25:25.245954 6988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:25.247181 6988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:25.248368 6988 hybrid_clock.cc:648] HybridClock initialized: now 1781958325248352 us; error 31 us; skew 500 ppm
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:25.251330 6988 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 12:25:25.252257 6988 webserver.cc:460] Webserver started at http://127.0.98.61:35195/ using document root <none> and password file <none>
I20260620 12:25:25.252840 6988 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.252930 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.253165 6988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:25.255555 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/instance:
uuid: "155e4e6405e544bba95e8774e9cbccb4"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "3b69f0eb383096f2eea5e6adb079c441"
server_key_iv: "5757d28f7ed0c33db26c70387d8894b3"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.256151 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal/instance:
uuid: "155e4e6405e544bba95e8774e9cbccb4"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "3b69f0eb383096f2eea5e6adb079c441"
server_key_iv: "5757d28f7ed0c33db26c70387d8894b3"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.259982 6988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:25:25.262434 7003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.263784 6988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:25.263926 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
uuid: "155e4e6405e544bba95e8774e9cbccb4"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "3b69f0eb383096f2eea5e6adb079c441"
server_key_iv: "5757d28f7ed0c33db26c70387d8894b3"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.264045 6988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.282482 6988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:25.286134 6988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:25.286350 6988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:25.293694 6988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33011
I20260620 12:25:25.294005 7055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33011 every 8 connection(s)
I20260620 12:25:25.294773 6988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:25:25.297082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 6988
I20260620 12:25:25.297873 7056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:25.301959 7056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:25.312264 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.298694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.0.98.61:33011)
0620 12:25:25.299157 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.299172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.299215 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:25:25.299874 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.299896 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.300243 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.300596 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.301418 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.302486 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.303389 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.304036 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.304206 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.306905 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.306911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.306923 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.306960 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.308940 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.309411 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.309417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.309424 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.309576 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.309923 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.309928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.309932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.310467 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.310707 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.311003 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.311285 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":141,"threads_started":1}
I20260620 12:25:25.313521 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.303761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.98.62:45973)
0620 12:25:25.303922 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.303927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.304230 (+ 303us) server_negotiation.cc:407] Connection header received
0620 12:25:25.304469 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.304472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.304530 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.304619 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.305292 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.305811 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.306899 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.307119 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.307197 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.309590 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.309593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.309595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.309653 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.311636 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.312097 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.312101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.312103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.312182 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.312513 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.312516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.312518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.312764 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.312833 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.313252 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.313369 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:25:25.313524 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.303701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:51328)
0620 12:25:25.304116 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.304166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.304315 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.304667 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.304673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.305038 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.305161 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.305167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.305941 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.305945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.306766 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.306775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.307236 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.307240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.307330 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.307789 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.307810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.309432 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.311750 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.311760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.311766 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.311994 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.312283 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.312286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.312288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.312402 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.312920 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.312926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.313036 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.313171 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.313334 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":147,"threads_started":1}
I20260620 12:25:25.314791 7056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:25:25.323519 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.315349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.0.98.61:33011)
0620 12:25:25.315457 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.315463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.315512 (+ 49us) server_negotiation.cc:407] Connection header received
0620 12:25:25.315637 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.315640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.315681 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.315755 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.316133 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.317039 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.318131 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.318449 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.318536 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.320616 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.320620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.320622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.320648 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.321923 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.322311 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.322317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.322319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.322388 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.322695 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.322698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.322699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.322872 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.322975 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.323210 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.323396 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":38}
I20260620 12:25:25.323519 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.315280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.0.1:37906)
0620 12:25:25.315455 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.315469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.315549 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.315776 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.315778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.315946 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.316023 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.316029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.317162 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.317165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.318019 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.318029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.318567 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.318571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.318689 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.319146 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.319160 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.320489 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.322023 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.322029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.322032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.322211 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.322483 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.322486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.322487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.322560 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.322973 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.322975 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.323063 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.323169 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.323325 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 12:25:25.326597 7056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Bootstrap starting.
I20260620 12:25:25.328868 7056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:25.329530 7056 log.cc:824] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:25.331435 7056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: No bootstrap required, opened a new log
I20260620 12:25:25.334304 7056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } }
I20260620 12:25:25.334514 7056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:25.334548 7056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155e4e6405e544bba95e8774e9cbccb4, State: Initialized, Role: FOLLOWER
I20260620 12:25:25.335033 7056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } }
I20260620 12:25:25.335438 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } } }
I20260620 12:25:25.335579 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:25:25.336261 7056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 12:25:25.340903 7073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:25:25.341044 7073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:25:25.342733 6988 master_runner.cc:428] Detected that this master 155e4e6405e544bba95e8774e9cbccb4 is joining an existing cluster
I20260620 12:25:25.342782 6988 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33011
I20260620 12:25:25.353438 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.345107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.0.1:37920)
0620 12:25:25.345360 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.345372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.345480 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.345714 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.345717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.345942 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.346041 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.346047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.347083 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.347087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.348019 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.348029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.348581 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.348584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.348696 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.349063 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.349102 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.349520 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.351746 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.351750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.351754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.352012 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.352317 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.352320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.352322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.352414 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.352861 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.352864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.352951 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.353093 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.353240 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260620 12:25:25.353535 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.344800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.0.98.61:33011)
0620 12:25:25.345033 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.345036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.345443 (+ 407us) server_negotiation.cc:407] Connection header received
0620 12:25:25.345570 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.345574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.345608 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.345691 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.346158 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.346972 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.348128 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.348474 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.348580 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.349644 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.349647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.349650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.349672 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.351622 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.352121 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.352125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.352127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.352212 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.352520 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.352526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.352527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.352731 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.352834 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.353136 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.353308 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":158}
I20260620 12:25:25.355057 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.344906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.0.98.62:45973)
0620 12:25:25.345036 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.345041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.345308 (+ 267us) server_negotiation.cc:407] Connection header received
0620 12:25:25.345504 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.345508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.345558 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.345612 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.346276 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.347081 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.347907 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.348181 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.348274 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.350054 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.350061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.350066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.350095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.352422 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.353208 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.353211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.353213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.353294 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.354126 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.354129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.354131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.354362 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.354480 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.354801 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.354913 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:25:25.355077 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.344718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:51330)
0620 12:25:25.345230 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.345244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.345381 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.345758 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.345760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.346014 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.346159 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.346166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.347181 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.347188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.347790 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.347798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.348292 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.348296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.348449 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.349318 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.349334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.349854 (+ 520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.352868 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.352871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.352873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.353093 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.353397 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.353400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.353402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.354014 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.354500 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.354506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.354607 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.354749 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.354896 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":26,"thread_start_us":101,"threads_started":1}
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:25:25.369493 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.359508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.0.98.61:33011)
0620 12:25:25.359613 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.359617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.360359 (+ 742us) server_negotiation.cc:407] Connection header received
0620 12:25:25.360529 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.360531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.360577 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.360646 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.361387 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.361850 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.363109 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.363307 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.363396 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.365876 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.365880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.365882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.365909 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.367416 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.368078 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.368081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.368083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.368167 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.368474 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.368477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.368479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.368714 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.368882 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.369233 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.369339 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:25:25.369532 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.359425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.0.1:37928)
0620 12:25:25.360224 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.360277 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.360408 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.360701 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.360708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.361136 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.361260 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.361267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.361952 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.361956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.362996 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.363008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.363424 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.363427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.363548 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.364031 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.364046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.365675 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.367550 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.367558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.367565 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.367956 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.368276 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.368279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.368282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.368377 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.368880 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.368888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.369020 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.369174 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.369377 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":88,"threads_started":1}
I20260620 12:25:25.370939 6524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33011: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33011
}
attrs {
promote: true
}
}
dest_uuid: "0c1403069df443ce8ec099e16da75a42"
cas_config_opid_index: -1
I20260620 12:25:25.371570 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } }
I20260620 12:25:25.374164 7030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:25.373600 7085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 155e4e6405e544bba95e8774e9cbccb4 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 12:25:25.375681 7030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Refusing update from remote peer 0c1403069df443ce8ec099e16da75a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 12:25:25.377506 7085 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 12:25:25.378085 6988 master.cc:537] Master@127.0.98.61:33011 shutting down...
I20260620 12:25:25.377609 7086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.002s
I20260620 12:25:25.377898 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } }
I20260620 12:25:25.379828 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
W20260620 12:25:25.381495 6511 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 12:25:25.382725 6988 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:25.383023 6988 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:25.383138 6988 tablet_replica.cc:330] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: stopping tablet replica
W20260620 12:25:25.383167 6511 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 -> Peer 155e4e6405e544bba95e8774e9cbccb4 (127.0.98.61:33011): Couldn't send request to peer 155e4e6405e544bba95e8774e9cbccb4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 12:25:25.398573 6988 master.cc:552] Master@127.0.98.61:33011 shutdown complete.
I20260620 12:25:25.398551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44753
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:25:25.398658 6988 master_runner.cc:305] Clearing existing system tablet
I20260620 12:25:25.398980 6988 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.399096 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.400290 6988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:25.401261 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.401468 6988 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.401605 6988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:25.401638 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
uuid: "155e4e6405e544bba95e8774e9cbccb4"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "3b69f0eb383096f2eea5e6adb079c441"
server_key_iv: "5757d28f7ed0c33db26c70387d8894b3"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.403898 6988 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:25.406294 6988 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 12:25:25.406383 6988 log.cc:1196] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 12:25:25.406627 6988 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Deleting consensus metadata
I20260620 12:25:25.406875 6988 master_runner.cc:315] Copying system tablet from 127.0.98.62:45973
I20260620 12:25:25.408066 6988 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45973
I20260620 12:25:25.417985 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.408778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.0.98.62:45973)
0620 12:25:25.408910 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.408915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.409188 (+ 273us) server_negotiation.cc:407] Connection header received
0620 12:25:25.409342 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.409345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.409398 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.409507 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.410067 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.410923 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.412124 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.412470 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.412584 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.414067 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.414072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.414075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.414103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.415850 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.416367 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.416371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.416373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.416463 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.416759 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.416762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.416764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.417300 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.417400 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.417693 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.417822 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:25:25.418339 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.408694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:51346)
0620 12:25:25.409069 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.409088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.409230 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.409545 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.409550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.409781 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.409928 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.409937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.411036 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.411040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.411991 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.412002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.412686 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.412692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.412791 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.413430 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.413444 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.413910 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.415977 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.415983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.415988 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.416262 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.416567 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.416570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.416572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.416663 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.417406 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.417409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.417517 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.417621 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.417770 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260620 12:25:25.419267 6544 tablet_copy_service.cc:140] P 0c1403069df443ce8ec099e16da75a42: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 155e4e6405e544bba95e8774e9cbccb4 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51346)
I20260620 12:25:25.419462 6544 tablet_copy_service.cc:161] P 0c1403069df443ce8ec099e16da75a42: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 155e4e6405e544bba95e8774e9cbccb4 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51346: session id = 155e4e6405e544bba95e8774e9cbccb4-00000000000000000000000000000000
I20260620 12:25:25.421361 6544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 12:25:25.423256 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 12:25:25.425648 6988 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: tablet copy: Starting download of 0 data blocks...
I20260620 12:25:25.425891 6988 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: tablet copy: Starting download of 1 WAL segments...
I20260620 12:25:25.427834 6988 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 12:25:25.429777 6544 tablet_copy_service.cc:342] P 0c1403069df443ce8ec099e16da75a42: Request end of tablet copy session 155e4e6405e544bba95e8774e9cbccb4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51346
I20260620 12:25:25.429915 6544 tablet_copy_service.cc:433] P 0c1403069df443ce8ec099e16da75a42: ending tablet copy session 155e4e6405e544bba95e8774e9cbccb4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 155e4e6405e544bba95e8774e9cbccb4
I20260620 12:25:25.437916 6988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:25.439728 7099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.439780 7100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.443763 7102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:25.446831 6988 server_base.cc:1034] running on GCE node
I20260620 12:25:25.447016 6988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:25.447324 6988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:25.448460 6988 hybrid_clock.cc:648] HybridClock initialized: now 1781958325448440 us; error 42 us; skew 500 ppm
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:25.450165 6988 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 12:25:25.450642 6988 webserver.cc:460] Webserver started at http://127.0.98.61:45191/ using document root <none> and password file <none>
I20260620 12:25:25.450870 6988 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.450912 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.452279 6988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:25.453322 7109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.453558 6988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:25.453701 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
uuid: "155e4e6405e544bba95e8774e9cbccb4"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "3b69f0eb383096f2eea5e6adb079c441"
server_key_iv: "5757d28f7ed0c33db26c70387d8894b3"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.453874 6988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.468873 6988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:25.471290 6988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:25.476025 6988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33011
I20260620 12:25:25.476069 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33011 every 8 connection(s)
I20260620 12:25:25.476747 6988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:25:25.478754 7162 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:25:25.479875 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Bootstrap starting.
I20260620 12:25:25.490871 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:25:25.491209 7162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: Bootstrap complete.
I20260620 12:25:25.491842 7162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } }
I20260620 12:25:25.491983 7162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 155e4e6405e544bba95e8774e9cbccb4, State: Initialized, Role: LEARNER
I20260620 12:25:25.492096 7162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } }
I20260620 12:25:25.492436 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } }
I20260620 12:25:25.492568 7162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:25.492609 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:25:25.496326 7176 catalog_manager.cc:1259] Loaded cluster ID: 4d6161b194794dc484afd4156351d26c
I20260620 12:25:25.496385 7176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: loading cluster ID for follower catalog manager: success
I20260620 12:25:25.498481 7176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: acquiring CA information for follower catalog manager: success
I20260620 12:25:25.499989 7176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 12:25:25.537933 7090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:25.538211 7090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:25.538292 7090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:25.542382 7090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:25.542443 7090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:25.542464 7090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:25.542521 7090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:25.542550 7090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:25.548170 7090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:25.549329 7090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:25.550426 7090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:25.559203 7184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.559208 7182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.559223 7181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:25.559247 7090 server_base.cc:1034] running on GCE node
I20260620 12:25:25.560040 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:25.561156 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:25.562347 7090 hybrid_clock.cc:648] HybridClock initialized: now 1781958325562328 us; error 43 us; skew 500 ppm
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:25.565268 7090 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 12:25:25.566138 7090 webserver.cc:460] Webserver started at http://127.0.98.60:36439/ using document root <none> and password file <none>
I20260620 12:25:25.566777 7090 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.566866 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.567106 7090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:25.569228 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/instance:
uuid: "92e4976b2677431ab33a7af4d2b0562b"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "161d932b77f076e3901845efbbeb0df7"
server_key_iv: "677155e9bdcf1f94880dab10c22a5428"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.569787 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal/instance:
uuid: "92e4976b2677431ab33a7af4d2b0562b"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "161d932b77f076e3901845efbbeb0df7"
server_key_iv: "677155e9bdcf1f94880dab10c22a5428"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.573660 7090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 12:25:25.576169 7191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.577418 7090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 12:25:25.577564 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
uuid: "92e4976b2677431ab33a7af4d2b0562b"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "161d932b77f076e3901845efbbeb0df7"
server_key_iv: "677155e9bdcf1f94880dab10c22a5428"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.577678 7090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.594256 7090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:25.597424 7090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:25.597643 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:25.602711 6715 heartbeater.cc:503] Master 127.0.98.62:45973 was elected leader, sending a full tablet report...
I20260620 12:25:25.605247 7090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44753
I20260620 12:25:25.605546 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44753 every 8 connection(s)
I20260620 12:25:25.606276 7090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:25:25.609026 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:25.612802 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:25:25.616081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 7090
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:25:25.627756 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.614802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.98.62:45973)
0620 12:25:25.614955 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.614960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.615773 (+ 813us) server_negotiation.cc:407] Connection header received
0620 12:25:25.616744 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.616748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.616806 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.616910 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.618797 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.619327 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.620466 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.620694 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.620806 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.623495 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.623498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.623500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.623525 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.625222 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.625761 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.625765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.625767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.625855 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.626297 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.626301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.626303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.626571 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.626643 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.627256 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.627443 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:25:25.628432 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.617016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.0.98.60:44753)
0620 12:25:25.617340 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.617350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.617373 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:25:25.617413 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.617420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.617652 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.617884 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.618696 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.619726 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.620532 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.620906 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.621071 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.623976 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.623984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.624000 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.624027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.625783 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.626295 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.626301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.626303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.626414 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.626725 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.626728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.626730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.626977 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.627130 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.627391 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.627562 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":231,"spinlock_wait_cycles":23424,"thread_start_us":132,"threads_started":1}
I20260620 12:25:25.628434 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.615160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:51348)
0620 12:25:25.615619 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.615663 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.616493 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.617091 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.617099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.617921 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.618602 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.618623 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.619459 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.619463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.620338 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.620346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.620841 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.620845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.621026 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.621645 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.621669 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.623319 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.625336 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.625343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.625351 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.625648 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.625953 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.625957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.625959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.626142 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.626693 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.626702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.626974 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.627170 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.627504 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":30,"thread_start_us":115,"threads_started":1}
I20260620 12:25:25.629645 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:25:25.637843 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.630157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.0.1:37934)
0620 12:25:25.630299 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.630313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.630377 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.630766 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.630768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.630890 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.630978 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.630983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.631844 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.631846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.632352 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.632358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.632759 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.632761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.632833 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.633231 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.633241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.634397 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.636439 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.636443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.636445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.636751 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.637063 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.637065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.637066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.637138 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.637466 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.637467 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.637541 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.637618 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.637714 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 12:25:25.637923 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.630279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.0.98.61:33011)
0620 12:25:25.630528 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.630533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.630546 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:25:25.630588 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.630590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.630638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.630751 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.631084 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.631756 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.632452 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.632669 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.632746 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.634618 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.634623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.634626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.634654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.636274 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.636866 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.636869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.636871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.636969 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.637215 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.637217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.637218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.637386 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.637460 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.637674 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.637772 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260620 12:25:25.638609 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:25:25.645951 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.639146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:44038)
0620 12:25:25.639252 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.639267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.639373 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.639540 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.639542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.639661 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.639726 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.639730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.640415 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.640418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.640919 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.640927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.641444 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.641446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.641503 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.641993 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.642005 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.643224 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.644722 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.644724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.644726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.644907 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.645144 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.645146 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.645147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.645216 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.645570 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.645572 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.645643 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.645724 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.645811 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 12:25:25.646013 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.639228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.0.98.60:44753)
0620 12:25:25.639342 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.639345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.639356 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:25:25.639430 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.639432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.639465 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.639527 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.639809 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.640288 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.641039 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.641278 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.641442 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.643383 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.643385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.643387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.643407 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.644617 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.644995 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.644997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.644998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.645064 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.645290 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.645292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.645293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.645483 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.645556 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.645779 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.645868 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:25:25.649581 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Bootstrap starting.
I20260620 12:25:25.651985 7244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:25.653133 7244 log.cc:824] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:25.655318 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: No bootstrap required, opened a new log
I20260620 12:25:25.658273 7244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 12:25:25.658492 7244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:25.658552 7244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e4976b2677431ab33a7af4d2b0562b, State: Initialized, Role: FOLLOWER
I20260620 12:25:25.659036 7244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 12:25:25.659505 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 12:25:25.659622 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:25:25.660192 7244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 12:25:25.664726 7261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:25:25.664816 7261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:25:25.668367 7090 master_runner.cc:428] Detected that this master 92e4976b2677431ab33a7af4d2b0562b is joining an existing cluster
I20260620 12:25:25.668434 7090 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44753
I20260620 12:25:25.680212 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.670592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.0.98.61:33011)
0620 12:25:25.670742 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.670748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.671087 (+ 339us) server_negotiation.cc:407] Connection header received
0620 12:25:25.671257 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.671260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.671326 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.671405 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.671975 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.672517 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.673705 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.673931 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.674018 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.675698 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.675702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.675705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.675733 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.678341 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.678870 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.678874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.678876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.678959 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.679257 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.679260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.679262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.679460 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.679538 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.679922 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.680045 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:25:25.680682 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.670537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.0.1:37942)
0620 12:25:25.671001 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.671020 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.671158 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.671450 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.671454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.671696 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.671858 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.671866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.672628 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.672632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.673584 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.673594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.674054 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.674058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.674159 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.674641 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.674685 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.675139 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.678463 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.678467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.678469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.678749 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.679065 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.679068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.679070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.679162 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.679592 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.679595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.679678 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.680354 (+ 676us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.680490 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"spinlock_wait_cycles":3456,"thread_start_us":147,"threads_started":1}
I20260620 12:25:25.682034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6586
I20260620 12:25:25.684100 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.670958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.0.98.60:44753)
0620 12:25:25.671069 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.671075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.674877 (+ 3802us) server_negotiation.cc:407] Connection header received
0620 12:25:25.675030 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.675035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.675073 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.675118 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.676333 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.676808 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.678209 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.678514 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.678589 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.679988 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.679992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.679994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.680016 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.681691 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.682979 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.682986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.682988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.683065 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.683346 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.683349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.683351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.683504 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.683599 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.683846 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.683979 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":39}
I20260620 12:25:25.684113 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.670900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:44050)
0620 12:25:25.674779 (+ 3879us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.674810 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.674935 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.675260 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.675265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.675739 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.675871 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.675879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.676913 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.676919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.678096 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.678102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.678759 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.678762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.678973 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.679522 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.679535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.679859 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.682670 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.682676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.682678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.682881 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.683170 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.683172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.683174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.683253 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.683599 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.683603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.683695 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.683791 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.683926 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3811,"mutex_wait_us":241,"spinlock_wait_cycles":1408,"thread_start_us":98,"threads_started":1}
I20260620 12:25:25.687940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39003
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45973
--webserver_port=36737
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:25:25.719759 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.670509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.98.62:45973)
0620 12:25:25.670655 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.670660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.674925 (+ 4265us) server_negotiation.cc:407] Connection header received
0620 12:25:25.675080 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.675083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.675133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.675181 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.676202 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.677009 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.678218 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.678552 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.678618 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.716330 (+ 37712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.716334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.716336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.716367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.717950 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.718420 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.718426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.718427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.718516 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.718876 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.718882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.718884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.719094 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.719205 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.719482 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.719599 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:25:25.719784 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.671152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:51352)
0620 12:25:25.674846 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.674861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.674979 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.675381 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.675383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.675545 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.676086 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.676095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.677117 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.677120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.678088 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.678098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.678766 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.678769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.678969 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.715324 (+ 36355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.715340 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.716088 (+ 748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.718110 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.718113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.718114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.718313 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.718613 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.718617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.718619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.718774 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.719194 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.719197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.719307 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.719413 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.719575 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3631,"mutex_wait_us":115,"spinlock_wait_cycles":31488,"thread_start_us":72,"threads_started":1}
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958324, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:25:25.731431 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.724152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.0.98.60:44753)
0620 12:25:25.724256 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.724261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.724327 (+ 66us) server_negotiation.cc:407] Connection header received
0620 12:25:25.724478 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.724481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.724532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.724613 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.725006 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.725498 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.726256 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.726483 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.726591 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.728495 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.728498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.728501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.728522 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.729873 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.730283 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.730298 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.730303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.730367 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.730632 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.730638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.730640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.730822 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.730928 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.731176 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.731275 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:25:25.731431 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.724109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:44058)
0620 12:25:25.724262 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.724276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.724366 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.724627 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.724630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.724819 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.724903 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.724908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.725602 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.725607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.726142 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.726149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.726590 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.726594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.726711 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.727158 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.727171 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.728368 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.729980 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.729987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.729989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.730185 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.730457 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.730461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.730463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.730546 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.730918 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.730921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.731018 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.731119 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.731260 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 12:25:25.732613 6524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44753: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44753
}
attrs {
promote: true
}
}
dest_uuid: "0c1403069df443ce8ec099e16da75a42"
cas_config_opid_index: 5
I20260620 12:25:25.733215 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } }
I20260620 12:25:25.734229 7085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 92e4976b2677431ab33a7af4d2b0562b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.734877 7218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:25.735720 7085 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44753
I20260620 12:25:25.735882 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.735981 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:25.736308 7218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 FOLLOWER]: Refusing update from remote peer 0c1403069df443ce8ec099e16da75a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 12:25:25.737320 7086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:25.738402 7090 master.cc:537] Master@127.0.98.60:44753 shutting down...
I20260620 12:25:25.741904 7218 raft_consensus.cc:858] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } }
I20260620 12:25:25.741999 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.737194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.0.98.61:33011)
0620 12:25:25.737332 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.737339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.737356 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:25:25.737405 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.737409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.737471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.737560 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:25.737985 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.738759 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.740558 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.741176 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.741256 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.741416 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.741773 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.741852 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:25:25.742498 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.736863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.0.1:37948)
0620 12:25:25.736949 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.736965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.737032 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.737759 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.737762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.737780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:25.737856 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.737863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.739252 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.739255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.740424 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.740433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.741513 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.741516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.741602 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.741647 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.742322 (+ 675us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.742390 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 12:25:25.743563 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 LEARNER]: Refusing update from remote peer 0c1403069df443ce8ec099e16da75a42: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 12:25:25.744001 7086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 12:25:25.745744 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 92e4976b2677431ab33a7af4d2b0562b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.747488 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c1403069df443ce8ec099e16da75a42. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.747597 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:25:25.747717 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.747812 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:25:25.748140 7085 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: attempting to promote NON_VOTER 155e4e6405e544bba95e8774e9cbccb4 to VOTER
I20260620 12:25:25.748564 7085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } }
I20260620 12:25:25.750551 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 LEARNER]: Refusing update from remote peer 0c1403069df443ce8ec099e16da75a42: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 12:25:25.751107 7086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 12:25:25.751219 6508 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 -> Peer 92e4976b2677431ab33a7af4d2b0562b (127.0.98.60:44753): Couldn't send request to peer 92e4976b2677431ab33a7af4d2b0562b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
W20260620 12:25:25.741986 7218 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } }
I20260620 12:25:25.753494 7218 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 155e4e6405e544bba95e8774e9cbccb4 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } }
I20260620 12:25:25.755744 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c1403069df443ce8ec099e16da75a42. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } }
I20260620 12:25:25.755872 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 12:25:25.755962 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } }
I20260620 12:25:25.756013 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 12:25:25.756067 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: true } } }
I20260620 12:25:25.756125 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 12:25:25.756107 7086 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 155e4e6405e544bba95e8774e9cbccb4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.758273 7090 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 12:25:25.758520 7090 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 12:25:25.758582 7090 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: stopping tablet replica
I20260620 12:25:25.758982 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.759125 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:25.759158 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.759253 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:25.759676 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 155e4e6405e544bba95e8774e9cbccb4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.761328 7273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.761451 7273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:25:25.776108 7090 master.cc:552] Master@127.0.98.60:44753 shutdown complete.
I20260620 12:25:25.776189 7090 master_runner.cc:305] Clearing existing system tablet
I20260620 12:25:25.776459 7090 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.776537 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.777642 7090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 12:25:25.778472 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.778729 7090 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.778896 7090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:25.778934 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
uuid: "92e4976b2677431ab33a7af4d2b0562b"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "161d932b77f076e3901845efbbeb0df7"
server_key_iv: "677155e9bdcf1f94880dab10c22a5428"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.780288 7090 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:25.782196 7090 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 12:25:25.782305 7090 log.cc:1196] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 12:25:25.782603 7090 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Deleting consensus metadata
I20260620 12:25:25.782843 7090 master_runner.cc:315] Copying system tablet from 127.0.98.62:45973
I20260620 12:25:25.783905 7090 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45973
I20260620 12:25:25.792338 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.784559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.98.62:45973)
0620 12:25:25.784688 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.784692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.784915 (+ 223us) server_negotiation.cc:407] Connection header received
0620 12:25:25.785081 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.785086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.785138 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.785232 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.785899 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.786591 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.787472 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.787749 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.787866 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.789057 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.789061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.789064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.789088 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.790703 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.791102 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.791106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.791108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.791197 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.791466 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.791471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.791475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.791657 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.791762 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.792080 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.792192 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:25:25.792353 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.784489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:51366)
0620 12:25:25.784818 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.784830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.784969 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.785261 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.785264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.785565 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.785747 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.785755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.786777 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.786780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.787349 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.787357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.787869 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.787872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.787955 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.788542 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.788554 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.788930 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.790793 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.790799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.790801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.791003 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.791291 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.791294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.791296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.791379 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.791751 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.791754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.791855 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.792019 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.792164 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":84,"threads_started":1}
I20260620 12:25:25.793099 6544 tablet_copy_service.cc:140] P 0c1403069df443ce8ec099e16da75a42: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 92e4976b2677431ab33a7af4d2b0562b ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51366)
I20260620 12:25:25.793191 6544 tablet_copy_service.cc:161] P 0c1403069df443ce8ec099e16da75a42: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 92e4976b2677431ab33a7af4d2b0562b at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51366: session id = 92e4976b2677431ab33a7af4d2b0562b-00000000000000000000000000000000
I20260620 12:25:25.794216 6544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 12:25:25.795401 7090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 12:25:25.797875 7090 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: tablet copy: Starting download of 0 data blocks...
I20260620 12:25:25.798079 7090 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: tablet copy: Starting download of 1 WAL segments...
I20260620 12:25:25.799405 7090 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 12:25:25.800872 6544 tablet_copy_service.cc:342] P 0c1403069df443ce8ec099e16da75a42: Request end of tablet copy session 92e4976b2677431ab33a7af4d2b0562b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51366
I20260620 12:25:25.800967 6544 tablet_copy_service.cc:433] P 0c1403069df443ce8ec099e16da75a42: ending tablet copy session 92e4976b2677431ab33a7af4d2b0562b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 92e4976b2677431ab33a7af4d2b0562b
I20260620 12:25:25.807623 7090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:25.809221 7288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.809247 7291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.809690 7289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:25.810206 7090 server_base.cc:1034] running on GCE node
I20260620 12:25:25.810367 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:25.810688 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:25.811832 7090 hybrid_clock.cc:648] HybridClock initialized: now 1781958325811824 us; error 24 us; skew 500 ppm
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:25.813365 7090 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 12:25:25.813767 7090 webserver.cc:460] Webserver started at http://127.0.98.60:45917/ using document root <none> and password file <none>
I20260620 12:25:25.813925 7090 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.813957 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.815553 7090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 12:25:25.816504 7298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.816707 7090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 12:25:25.816771 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
uuid: "92e4976b2677431ab33a7af4d2b0562b"
format_stamp: "Formatted at 2026-06-20 12:25:25 on dist-test-slave-bwpp"
server_key: "161d932b77f076e3901845efbbeb0df7"
server_key_iv: "677155e9bdcf1f94880dab10c22a5428"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.816845 7090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.835347 6849 heartbeater.cc:503] Master 127.0.98.62:45973 was elected leader, sending a full tablet report...
W20260620 12:25:25.841025 7270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:25.841427 7270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:25.841508 7270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:25.847441 7270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:25.847520 7270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:25.847607 7270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:25.855363 7270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36737
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:25.856691 7270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:25.858000 7270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:25.870898 7307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:25.871294 7308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:25.871340 7090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260620 12:25:25.871353 7310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:25.872027 7270 server_base.cc:1034] running on GCE node
I20260620 12:25:25.872810 7270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:25.873590 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:25.873857 7270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:25.875061 7270 hybrid_clock.cc:648] HybridClock initialized: now 1781958325874975 us; error 97 us; skew 500 ppm
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:25.878464 7090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44753
I20260620 12:25:25.878928 7270 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:25:25.879094 7360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44753 every 8 connection(s)
I20260620 12:25:25.879213 7090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:25:25.879918 7270 webserver.cc:460] Webserver started at http://127.0.98.1:36737/ using document root <none> and password file <none>
I20260620 12:25:25.880527 7270 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:25.880633 7270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:25.880831 7363 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:25:25.881608 7363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Bootstrap starting.
I20260620 12:25:25.885178 7270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 12:25:25.887384 7368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.888803 7270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 12:25:25.889179 7363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:25:25.889432 7363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Bootstrap complete.
I20260620 12:25:25.888939 7270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "f245e2fae6a64ed49ca6086fa18cb731"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "15ef800908e370a82e7eccdaf7bde3c2"
server_key_iv: "0f1a7cc95290231bb45e936cb783883b"
server_key_version: "encryptionkey@0"
I20260620 12:25:25.889588 7270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:25.890044 7363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } }
I20260620 12:25:25.890142 7363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92e4976b2677431ab33a7af4d2b0562b, State: Initialized, Role: LEARNER
I20260620 12:25:25.890250 7363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } }
I20260620 12:25:25.890551 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:25.890692 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: LEARNER
I20260620 12:25:25.890745 7363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:25.893918 7380 catalog_manager.cc:1259] Loaded cluster ID: 4d6161b194794dc484afd4156351d26c
I20260620 12:25:25.893976 7380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: loading cluster ID for follower catalog manager: success
I20260620 12:25:25.895761 7380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: acquiring CA information for follower catalog manager: success
I20260620 12:25:25.896777 7380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:25:25.904639 7270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:25.907603 7270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:25.907797 7270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:25.908376 7270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:25.909224 7270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:25.909291 7270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.909363 7270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:25.909415 7270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:25.918040 7270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39003
I20260620 12:25:25.918207 7493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39003 every 8 connection(s)
I20260620 12:25:25.919195 7270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:25.926396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 7270
I20260620 12:25:25.926604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6720
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 12:25:25 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958325, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:25:25.935724 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.921045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35925 (local address 127.0.98.62:45973)
0620 12:25:25.921180 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.921185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.922213 (+ 1028us) server_negotiation.cc:407] Connection header received
0620 12:25:25.923351 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.923354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.923412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.923466 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.924699 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.925485 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.926712 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.927033 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.927099 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.931307 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.931314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.931319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.931349 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.933168 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.933836 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.933840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.933842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.933925 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.934352 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.934355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.934357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.934573 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.934674 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.935408 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.935555 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 12:25:25.935791 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.920975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45907 (local address 127.0.98.61:33011)
0620 12:25:25.922354 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.922358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.922443 (+ 85us) server_negotiation.cc:407] Connection header received
0620 12:25:25.923335 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.923339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.923399 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.923483 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.924678 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.925199 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.926424 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.926826 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.926919 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.931652 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.931656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.931661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.931693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.933394 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.933946 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.933953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.933958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.934055 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.934363 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.934368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.934373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.934604 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.934710 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.935413 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.935550 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291}
I20260620 12:25:25.936539 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.922363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35733 (local address 127.0.98.60:44753)
0620 12:25:25.922596 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:25.922600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:25.922785 (+ 185us) server_negotiation.cc:407] Connection header received
0620 12:25:25.923427 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:25.923430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:25.923485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:25.923561 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:25.924699 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.925731 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.926920 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.927252 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.927340 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.931026 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:25.931030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:25.931033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:25.931064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:25.933053 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.933836 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.933840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.933842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.933925 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:25.934352 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:25.934355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:25.934357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:25.934570 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:25.934678 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:25.936257 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0620 12:25:25.936384 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260620 12:25:25.937106 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.921632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.98.1:35733)
0620 12:25:25.922645 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.922703 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.923161 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.923669 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.923672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.924106 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.924558 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.924568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.925842 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.925845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.926809 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.926817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.927455 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.927459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.927573 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.928234 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.928255 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.930768 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.933190 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.933199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.933217 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.933720 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.934036 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.934039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.934041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.934241 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.934731 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.934737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.935126 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.935333 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.935693 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":138,"spinlock_wait_cycles":25856,"thread_start_us":94,"threads_started":1}
I20260620 12:25:25.937363 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.921364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.98.1:45907)
0620 12:25:25.922365 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.922378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.923143 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.923647 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.923654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.924033 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.924543 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.924556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.925334 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.925337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.926278 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.926290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.927129 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.927136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.927390 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.928325 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.928346 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.931497 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.933516 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.933523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.933528 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.933837 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.934168 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.934174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.934179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.934268 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.934785 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.934791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.935127 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.936056 (+ 929us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.936219 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":158,"spinlock_wait_cycles":21120,"thread_start_us":124,"threads_started":1}
I20260620 12:25:25.937659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39413
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45973
--webserver_port=44375
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:25:25.938686 7494 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44753
I20260620 12:25:25.938910 7494 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:25.937121 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:25.921364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:35925)
0620 12:25:25.922044 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:25.922109 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:25.923161 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:25.923658 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:25.923661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:25.924106 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:25.924561 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.924568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.925627 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.925629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.926553 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:25.926563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:25.927242 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:25.927245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:25.927398 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:25.928247 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:25.928260 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:25.931148 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:25.933272 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.933277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.933279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.933720 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.934036 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:25.934039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:25.934041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:25.934241 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:25.934731 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:25.934737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:25.935126 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 12:25:25.935890 (+ 764us) client_negotiation.cc:240] Negotiation successful
0620 12:25:25.936070 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":189,"spinlock_wait_cycles":19840,"thread_start_us":708,"threads_started":1}
I20260620 12:25:25.939523 7494 heartbeater.cc:511] Master 127.0.98.60:44753 requested a full tablet report, sending...
I20260620 12:25:25.939791 7495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260620 12:25:25.939891 7495 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:25.940068 7495 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260620 12:25:25.940855 6524 ts_manager.cc:194] Re-registered known tserver with Master: f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1:39003)
I20260620 12:25:25.941118 7325 ts_manager.cc:194] Registered new tserver with Master: f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1:39003)
I20260620 12:25:25.941455 7496 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33011
I20260620 12:25:25.941555 7496 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:25.941557 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35925
I20260620 12:25:25.941712 7496 heartbeater.cc:511] Master 127.0.98.61:33011 requested a full tablet report, sending...
I20260620 12:25:25.942740 7126 ts_manager.cc:194] Registered new tserver with Master: f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1:39003)
W20260620 12:25:26.055778 7502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:26.056066 7502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:26.056149 7502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:26.060328 7502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:26.060423 7502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:26.060523 7502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:26.066120 7502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44375
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:26.067320 7502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:26.068347 7502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:25:26.077605 6983 heartbeater.cc:503] Master 127.0.98.62:45973 was elected leader, sending a full tablet report...
W20260620 12:25:26.078444 7507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:26.078397 7508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:26.078384 7510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:26.078905 7502 server_base.cc:1034] running on GCE node
I20260620 12:25:26.079371 7502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:26.080030 7502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:26.081209 7502 hybrid_clock.cc:648] HybridClock initialized: now 1781958326081195 us; error 35 us; skew 500 ppm
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:26.084172 7502 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:25:26.085040 7502 webserver.cc:460] Webserver started at http://127.0.98.2:44375/ using document root <none> and password file <none>
I20260620 12:25:26.085619 7502 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:26.085717 7502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:26.089795 7502 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:25:26.092206 7517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:26.093855 7502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 12:25:26.093953 7502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "fe7a58f517d04d10aa1f256aa931244e"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3056e837a0939f82486b5c71dd75d624"
server_key_iv: "dbc2628541cf6a17c9589e064c5c5a88"
server_key_version: "encryptionkey@0"
I20260620 12:25:26.094728 7502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:26.106913 7502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:26.110203 7502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:26.110415 7502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:26.111044 7502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:26.111907 7502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:26.111953 7502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:26.112020 7502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:26.112049 7502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:26.120543 7502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39413
I20260620 12:25:26.120596 7630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39413 every 8 connection(s)
I20260620 12:25:26.121611 7502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:26.125903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 7502
I20260620 12:25:26.126111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6854
I20260620 12:25:26.133694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46617
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45973
--webserver_port=35613
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:26.140200 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.123699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51955 (local address 127.0.98.62:45973)
0620 12:25:26.123935 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.123939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.125365 (+ 1426us) server_negotiation.cc:407] Connection header received
0620 12:25:26.126685 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.126691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.126758 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.126816 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.130775 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.131529 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.132618 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.132958 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.133039 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.135853 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.135857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.135860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.135891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.137988 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138599 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.138602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.138604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.138715 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.139039 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.139046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.139048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.139260 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.139340 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.139897 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.140027 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260620 12:25:26.140200 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.123813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33441 (local address 127.0.98.60:44753)
0620 12:25:26.123937 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.123941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.126742 (+ 2801us) server_negotiation.cc:407] Connection header received
0620 12:25:26.126917 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.126923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.126980 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.127031 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.129065 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.129575 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.130861 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.131236 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.131312 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.135853 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.135857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.135860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.135891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.137947 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138600 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.138603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.138605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.138715 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.139061 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.139064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.139066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.139260 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.139340 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.139897 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.140027 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:25:26.140316 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.123549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41217 (local address 127.0.98.61:33011)
0620 12:25:26.123844 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.123848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.127843 (+ 3995us) server_negotiation.cc:407] Connection header received
0620 12:25:26.127903 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.127909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.127970 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.128054 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.128791 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.129723 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.131191 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.131622 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.131714 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.135628 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.135634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.135639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.135674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.137562 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138213 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.138220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.138224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.138311 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138764 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.138771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.138776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.139051 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.139130 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.139920 (+ 790us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.140078 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260620 12:25:26.142071 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.124517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.98.2:33441)
0620 12:25:26.126643 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.126681 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.126810 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.127164 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.127173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.128777 (+ 1604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:26.128927 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.128937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.129725 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.129728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.130722 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.130735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.131505 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.131508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.131730 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.132597 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:26.132619 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:26.135700 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:26.138069 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.138080 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.138481 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.138842 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.138847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.138958 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.139400 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:26.139403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:26.139586 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.140378 (+ 792us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.140979 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2054,"mutex_wait_us":287,"spinlock_wait_cycles":26624,"thread_start_us":87,"threads_started":1}
I20260620 12:25:26.142073 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.124050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.98.2:41217)
0620 12:25:26.126258 (+ 2208us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.126272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.126356 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.128216 (+ 1860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.128219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.128414 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:26.128602 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.128625 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.129882 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.129888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.131048 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.131062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.131893 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.131899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.132036 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.132674 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:26.132695 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:26.135267 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:26.137695 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.137702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.137723 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.138097 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.138428 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.138439 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.138626 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.139217 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:26.139227 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:26.139588 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.140534 (+ 946us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.140826 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2111,"mutex_wait_us":30,"spinlock_wait_cycles":12032,"thread_start_us":175,"threads_started":1}
I20260620 12:25:26.142716 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.124049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:51955)
0620 12:25:26.125222 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.125262 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.125993 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.127164 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.127173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.127873 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:26.128603 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.128626 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.131712 (+ 3086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.131715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.132447 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.132457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.133207 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.133213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.133350 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.133905 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:26.133925 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:26.135681 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:26.138125 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.138135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.138481 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.138837 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.138840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.138841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.138928 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.139403 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:26.139405 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:26.139586 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.140470 (+ 884us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.140925 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":217,"spinlock_wait_cycles":12160,"thread_start_us":135,"threads_started":1}
I20260620 12:25:26.143466 7631 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44753
I20260620 12:25:26.143501 7632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260620 12:25:26.143873 7631 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:26.143873 7632 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:26.144474 7632 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260620 12:25:26.145159 7631 heartbeater.cc:511] Master 127.0.98.60:44753 requested a full tablet report, sending...
I20260620 12:25:26.145721 6524 ts_manager.cc:194] Re-registered known tserver with Master: fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413)
I20260620 12:25:26.145833 7325 ts_manager.cc:194] Registered new tserver with Master: fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413)
I20260620 12:25:26.146432 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51955
I20260620 12:25:26.148488 7633 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33011
I20260620 12:25:26.148582 7633 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:26.148783 7633 heartbeater.cc:511] Master 127.0.98.61:33011 requested a full tablet report, sending...
I20260620 12:25:26.151067 7126 ts_manager.cc:194] Registered new tserver with Master: fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413)
W20260620 12:25:26.254880 7638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:26.255136 7638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:26.255180 7638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:26.259131 7638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:26.259186 7638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:26.259243 7638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:26.264559 7638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35613
--tserver_master_addrs=127.0.98.62:45973,127.0.98.61:33011,127.0.98.60:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:26.265620 7638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:26.266796 7638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:26.276743 7644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:26.276731 7643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:26.276870 7638 server_base.cc:1034] running on GCE node
W20260620 12:25:26.276731 7646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:26.277436 7638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:26.278046 7638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:26.279229 7638 hybrid_clock.cc:648] HybridClock initialized: now 1781958326279216 us; error 44 us; skew 500 ppm
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:26.282064 7638 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:25:26.283020 7638 webserver.cc:460] Webserver started at http://127.0.98.3:35613/ using document root <none> and password file <none>
I20260620 12:25:26.283581 7638 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:26.283699 7638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:26.288146 7638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 12:25:26.290496 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.286379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.0.98.60:44753)
0620 12:25:26.286523 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.286528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.286631 (+ 103us) server_negotiation.cc:407] Connection header received
0620 12:25:26.286821 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.286824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.286886 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.286968 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:26.287261 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.288031 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.289281 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.289868 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.289977 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.290143 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.290313 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.290389 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:25:26.290542 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.286290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:44060)
0620 12:25:26.286542 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.286555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.286696 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.287002 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.287005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.287025 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:26.287145 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.287152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.288145 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.288148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.289158 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.289169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.289993 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.289996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.290167 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.290195 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.290271 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.290398 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260620 12:25:26.290607 7655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:26.291239 7335 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 12:25:26.291540 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c1403069df443ce8ec099e16da75a42. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:26.291677 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: LEARNER
I20260620 12:25:26.292505 7638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 12:25:26.292599 7638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "a440174e3fc34439b319063e963f916a"
format_stamp: "Formatted at 2026-06-20 12:25:24 on dist-test-slave-bwpp"
server_key: "3d9b3ff476571786b1f1908f12310f98"
server_key_iv: "530482b373b2417ba3e4440c2b8b1953"
server_key_version: "encryptionkey@0"
I20260620 12:25:26.293116 7638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:26.313987 7638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:26.317703 7638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:26.317914 7638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:26.318598 7638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:26.319516 7638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:26.319563 7638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:26.319631 7638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:26.319661 7638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:26.328480 7638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46617
I20260620 12:25:26.328800 7768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46617 every 8 connection(s)
I20260620 12:25:26.329583 7638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:26.333194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 7638
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 12:25:26 dist-test-slave-bwpp krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958326, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:26.346923 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.331395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40245 (local address 127.0.98.60:44753)
0620 12:25:26.331558 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.331562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.332970 (+ 1408us) server_negotiation.cc:407] Connection header received
0620 12:25:26.333830 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.333834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.333889 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.333942 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.335917 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.336760 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.338907 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.339226 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.339300 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.342607 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.342612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.342616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.342642 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.344400 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.345042 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.345046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.345049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.345131 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.345556 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.345560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.345562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.345794 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.345873 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.346582 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.346725 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:25:26.347127 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.332870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51097 (local address 127.0.98.61:33011)
0620 12:25:26.333022 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.333027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.333256 (+ 229us) server_negotiation.cc:407] Connection header received
0620 12:25:26.333811 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.333815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.333869 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.333926 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.336217 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.336934 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.338904 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.339225 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.339299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.343126 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.343131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.343133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.343160 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.345292 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.345750 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.345754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.345756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.345838 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.346142 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.346145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.346150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.346378 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.346460 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.346826 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.346941 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:25:26.348184 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.331400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51279 (local address 127.0.98.62:45973)
0620 12:25:26.331639 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.331642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.332535 (+ 893us) server_negotiation.cc:407] Connection header received
0620 12:25:26.333825 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.333829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.333889 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.333962 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.335950 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.336745 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.339652 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.340030 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.340106 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.343961 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.343966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.343970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.344001 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.346445 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.346925 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.346928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.346930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.347011 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.347310 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.347313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.347315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.347525 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.347609 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.347937 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.348054 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260620 12:25:26.348663 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.331849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:51279)
0620 12:25:26.332377 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.332437 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.333573 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.334337 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.334347 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.335004 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:26.335778 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.335788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.336933 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.336937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.339530 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.339537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.340282 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.340287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.340406 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.342321 (+ 1915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:26.342336 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:26.343786 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:26.346546 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.346548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.346550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.346810 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.347116 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.347119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.347120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.347203 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.347663 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:26.347666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:26.347745 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.348357 (+ 612us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.348480 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":1230,"thread_start_us":140,"threads_started":1}
I20260620 12:25:26.348672 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.331849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.98.3:40245)
0620 12:25:26.332866 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.332878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.333580 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.334337 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.334346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.335007 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:26.335758 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.335780 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.336936 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.336938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.338768 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.338779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.339492 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.339496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.339685 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.340483 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:26.340503 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:26.342417 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:26.344539 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.344546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.344562 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.344916 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.345249 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.345253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.345256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.345448 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.345962 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:26.345968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:26.346273 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.347149 (+ 876us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.347676 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":586,"spinlock_wait_cycles":76032,"thread_start_us":137,"threads_started":1}
I20260620 12:25:26.349212 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.332077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.98.3:51097)
0620 12:25:26.333157 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.333167 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.333580 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.334337 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.334346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.335968 (+ 1622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:26.336079 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.336086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.337086 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.337089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.338768 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.338782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.339493 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.339500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.339684 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.340749 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:26.340775 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:26.342973 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:26.345393 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.345396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.345398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.345642 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.345950 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:26.345953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:26.345955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:26.346040 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:26.346522 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:26.346527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:26.346618 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.347271 (+ 653us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.347577 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":301,"thread_start_us":130,"threads_started":1}
I20260620 12:25:26.349929 7770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45973
I20260620 12:25:26.350041 7771 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33011
I20260620 12:25:26.350167 7770 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:26.350167 7771 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:26.350287 7769 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44753
I20260620 12:25:26.350690 7769 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:26.350790 7770 heartbeater.cc:511] Master 127.0.98.62:45973 requested a full tablet report, sending...
I20260620 12:25:26.351089 7769 heartbeater.cc:511] Master 127.0.98.60:44753 requested a full tablet report, sending...
I20260620 12:25:26.350790 7771 heartbeater.cc:511] Master 127.0.98.61:33011 requested a full tablet report, sending...
I20260620 12:25:26.351972 6524 ts_manager.cc:194] Re-registered known tserver with Master: a440174e3fc34439b319063e963f916a (127.0.98.3:46617)
I20260620 12:25:26.352056 7325 ts_manager.cc:194] Registered new tserver with Master: a440174e3fc34439b319063e963f916a (127.0.98.3:46617)
I20260620 12:25:26.352356 7126 ts_manager.cc:194] Registered new tserver with Master: a440174e3fc34439b319063e963f916a (127.0.98.3:46617)
I20260620 12:25:26.352617 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51279
I20260620 12:25:26.355800 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.336030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.0.98.60:44753)
0620 12:25:26.337009 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.337014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.339264 (+ 2250us) server_negotiation.cc:407] Connection header received
0620 12:25:26.339304 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.339307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.339354 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.339430 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.341482 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.342125 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.344819 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.345194 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.345271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.348041 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.348045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.348047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.348070 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.353002 (+ 4932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.354186 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.354192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.354194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.354287 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.354696 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.354700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.354701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.354950 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.355124 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.355334 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.355591 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":79,"threads_started":1}
I20260620 12:25:26.356225 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.335228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.98.62:45973)
0620 12:25:26.337784 (+ 2556us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.337797 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.337811 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:26.337848 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.337852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.337900 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.337995 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.339029 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.339770 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.340886 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.341260 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.341353 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.343817 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.343824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.343829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.343857 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.354115 (+ 10258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.354585 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.354590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.354592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.354700 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.354992 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.354998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.355000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.355207 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.355329 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.355942 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.356077 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2488,"thread_start_us":90,"threads_started":1}
I20260620 12:25:26.358742 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.335635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.98.61:33011)
0620 12:25:26.344341 (+ 8706us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.344345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.344357 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:25:26.344388 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.344391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.344429 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.345553 (+ 1124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.349622 (+ 4069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.350365 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.351498 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.352984 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.353441 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.354727 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.354731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.354733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.354761 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.357070 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.357492 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.357495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.357496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.357567 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.357867 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.357870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.357872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.358038 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.358163 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.358396 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.358514 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8637,"thread_start_us":68,"threads_started":1}
I20260620 12:25:26.359236 6524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:26.362146 6524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:26.364686 7649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: true } } }
I20260620 12:25:26.364822 7649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:26.366271 7787 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: attempting to promote NON_VOTER 92e4976b2677431ab33a7af4d2b0562b to VOTER
I20260620 12:25:26.366647 7787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } }
I20260620 12:25:26.367800 7335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 LEARNER]: Refusing update from remote peer 0c1403069df443ce8ec099e16da75a42: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 12:25:26.367834 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Refusing update from remote peer 0c1403069df443ce8ec099e16da75a42: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 12:25:26.368137 7788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 12:25:26.368273 7649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 12:25:26.369974 7787 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 92e4976b2677431ab33a7af4d2b0562b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.370425 7335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 92e4976b2677431ab33a7af4d2b0562b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.370461 7136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 92e4976b2677431ab33a7af4d2b0562b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.371541 7787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.371673 7787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:26.371709 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.371819 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:25:26.371837 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.371927 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:26.371973 7649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.372081 7649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:26.372269 7798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c1403069df443ce8ec099e16da75a42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:26.372399 7798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:25:26.385178 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.379784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46617 (local address 127.0.0.1:43286)
0620 12:25:26.379908 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.379926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.380005 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.380838 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.380843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.380860 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:26.380974 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.380980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.382569 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.382573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.383656 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.383665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.384759 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.384763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.384848 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.384873 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.384952 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.385026 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260620 12:25:26.385207 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.379983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.0.98.3:46617)
0620 12:25:26.380289 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.380303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.380327 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:25:26.380408 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.380413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.380670 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.381140 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:26.381272 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.382457 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.383778 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.384648 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.384750 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.384895 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.384977 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.385065 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":133,"threads_started":1}
I20260620 12:25:26.385854 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.380782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39003 (local address 127.0.0.1:33342)
0620 12:25:26.381015 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.381028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.381099 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.382102 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.382105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.382121 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:26.382200 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.382205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.383594 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.383598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.384434 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.384443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.385474 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.385478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.385576 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.385602 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.385661 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.385724 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260620 12:25:26.386377 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.381183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.0.98.1:39003)
0620 12:25:26.381653 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.381665 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.381688 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:25:26.381759 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.381764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.381953 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.382325 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:26.382445 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.383472 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.384556 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.385315 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.386031 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.386165 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.386206 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.386271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
I20260620 12:25:26.386420 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.380043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39413 (local address 127.0.0.1:41364)
0620 12:25:26.380747 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.380761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.380855 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.381557 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.381560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.381578 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:26.381709 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.381716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.383082 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.383085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.384285 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.384295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.386080 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.386083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.386168 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.386188 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.386245 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.386304 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":23,"thread_start_us":105,"threads_started":1}
I20260620 12:25:26.387208 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.380386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.0.98.2:39413)
0620 12:25:26.380759 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.380777 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.380817 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:25:26.381024 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.381031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.381237 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.381397 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:26.381859 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.382962 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.384413 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.385749 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.385883 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.386929 (+ 1046us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.386975 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.387063 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":87,"threads_started":1}
I20260620 12:25:26.388877 7703 tablet_service.cc:1467] Processing CreateTablet for tablet 914f118a944143c5ae1eb4ed2e5a02f9 (DEFAULT_TABLE table=test-table [id=1a87bb84b46541c19f184c572958eaf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:26.388991 7428 tablet_service.cc:1467] Processing CreateTablet for tablet 914f118a944143c5ae1eb4ed2e5a02f9 (DEFAULT_TABLE table=test-table [id=1a87bb84b46541c19f184c572958eaf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:26.389986 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 914f118a944143c5ae1eb4ed2e5a02f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:26.389995 7703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 914f118a944143c5ae1eb4ed2e5a02f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:26.392860 7565 tablet_service.cc:1467] Processing CreateTablet for tablet 914f118a944143c5ae1eb4ed2e5a02f9 (DEFAULT_TABLE table=test-table [id=1a87bb84b46541c19f184c572958eaf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:26.393965 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 914f118a944143c5ae1eb4ed2e5a02f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:26.398898 7804 tablet_bootstrap.cc:492] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Bootstrap starting.
I20260620 12:25:26.399721 7805 tablet_bootstrap.cc:492] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Bootstrap starting.
I20260620 12:25:26.401582 7805 tablet_bootstrap.cc:654] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:26.401679 7804 tablet_bootstrap.cc:654] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:26.402408 7805 log.cc:824] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:26.402438 7806 tablet_bootstrap.cc:492] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Bootstrap starting.
I20260620 12:25:26.402457 7804 log.cc:824] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:26.404408 7806 tablet_bootstrap.cc:654] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:26.404435 7805 tablet_bootstrap.cc:492] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: No bootstrap required, opened a new log
I20260620 12:25:26.404651 7805 ts_tablet_manager.cc:1397] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:26.405186 7806 log.cc:824] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:26.406903 7806 tablet_bootstrap.cc:492] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: No bootstrap required, opened a new log
I20260620 12:25:26.407094 7806 ts_tablet_manager.cc:1397] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:26.408025 7804 tablet_bootstrap.cc:492] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: No bootstrap required, opened a new log
I20260620 12:25:26.408232 7804 ts_tablet_manager.cc:1397] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 12:25:26.409437 7805 raft_consensus.cc:348] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.409739 7805 raft_consensus.cc:374] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:26.409794 7805 raft_consensus.cc:729] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a440174e3fc34439b319063e963f916a, State: Initialized, Role: FOLLOWER
I20260620 12:25:26.410326 7805 consensus_queue.cc:260] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.410482 7806 raft_consensus.cc:348] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.410761 7806 raft_consensus.cc:374] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:26.410840 7806 raft_consensus.cc:729] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7a58f517d04d10aa1f256aa931244e, State: Initialized, Role: FOLLOWER
I20260620 12:25:26.411281 7806 consensus_queue.cc:260] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.411386 7805 ts_tablet_manager.cc:1428] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260620 12:25:26.411592 7770 heartbeater.cc:503] Master 127.0.98.62:45973 was elected leader, sending a full tablet report...
I20260620 12:25:26.412216 7806 ts_tablet_manager.cc:1428] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:25:26.412492 7632 heartbeater.cc:503] Master 127.0.98.62:45973 was elected leader, sending a full tablet report...
I20260620 12:25:26.417421 7804 raft_consensus.cc:348] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.417659 7804 raft_consensus.cc:374] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:26.417728 7804 raft_consensus.cc:729] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f245e2fae6a64ed49ca6086fa18cb731, State: Initialized, Role: FOLLOWER
I20260620 12:25:26.418216 7804 consensus_queue.cc:260] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.418851 7495 heartbeater.cc:503] Master 127.0.98.62:45973 was elected leader, sending a full tablet report...
I20260620 12:25:26.419149 7804 ts_tablet_manager.cc:1428] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
W20260620 12:25:26.423401 7497 tablet.cc:2369] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:25:26.581015 7772 tablet.cc:2369] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:25:26.623099 7634 tablet.cc:2369] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:26.659739 7810 raft_consensus.cc:482] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:26.659976 7810 raft_consensus.cc:504] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.661181 7810 leader_election.cc:288] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413), f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1:39003)
I20260620 12:25:26.664887 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.661544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35421 (local address 127.0.98.2:39413)
0620 12:25:26.661686 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.661692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.661708 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:25:26.661904 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.661908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.661965 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.662042 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:26.662313 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.662894 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.663946 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.664344 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.664459 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.664527 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.664689 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.664761 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:25:26.664908 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.661460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39413 (local address 127.0.98.3:35421)
0620 12:25:26.661604 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.661622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.661733 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.662066 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.662069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.662086 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:26.662203 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.662208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.663002 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.663007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.663824 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.663833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.664459 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.664463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.664560 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.664588 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.664674 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.664749 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260620 12:25:26.665096 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.661924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57477 (local address 127.0.98.1:39003)
0620 12:25:26.662085 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.662092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.662107 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:26.662161 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.662163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.662200 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.662279 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:26.662541 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.663102 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.664109 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.664505 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.664619 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.664677 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.664855 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.664928 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:25:26.665436 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.661763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39003 (local address 127.0.98.3:57477)
0620 12:25:26.661893 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:26.661907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:26.662019 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:26.662287 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:26.662290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:26.662306 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:26.662424 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.662429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.663210 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.663216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.663974 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:26.663982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.664620 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.664623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:26.664721 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.664749 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:25:26.664839 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:25:26.664906 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":23}
I20260620 12:25:26.665580 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "914f118a944143c5ae1eb4ed2e5a02f9" candidate_uuid: "a440174e3fc34439b319063e963f916a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a58f517d04d10aa1f256aa931244e" is_pre_election: true
I20260620 12:25:26.665966 7585 raft_consensus.cc:2454] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a440174e3fc34439b319063e963f916a in term 0.
I20260620 12:25:26.666096 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "914f118a944143c5ae1eb4ed2e5a02f9" candidate_uuid: "a440174e3fc34439b319063e963f916a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f245e2fae6a64ed49ca6086fa18cb731" is_pre_election: true
I20260620 12:25:26.666389 7448 raft_consensus.cc:2454] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a440174e3fc34439b319063e963f916a in term 0.
I20260620 12:25:26.666456 7659 leader_election.cc:302] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a440174e3fc34439b319063e963f916a, fe7a58f517d04d10aa1f256aa931244e; no voters:
I20260620 12:25:26.666862 7810 raft_consensus.cc:2781] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:26.666956 7810 raft_consensus.cc:482] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:26.667004 7810 raft_consensus.cc:3037] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:26.668054 7810 raft_consensus.cc:504] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.668422 7810 leader_election.cc:288] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [CANDIDATE]: Term 1 election: Requested vote from peers fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413), f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1:39003)
I20260620 12:25:26.668792 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "914f118a944143c5ae1eb4ed2e5a02f9" candidate_uuid: "a440174e3fc34439b319063e963f916a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a58f517d04d10aa1f256aa931244e"
I20260620 12:25:26.668799 7448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "914f118a944143c5ae1eb4ed2e5a02f9" candidate_uuid: "a440174e3fc34439b319063e963f916a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f245e2fae6a64ed49ca6086fa18cb731"
I20260620 12:25:26.668920 7448 raft_consensus.cc:3037] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:26.668920 7585 raft_consensus.cc:3037] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:26.670146 7585 raft_consensus.cc:2454] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a440174e3fc34439b319063e963f916a in term 1.
I20260620 12:25:26.670146 7448 raft_consensus.cc:2454] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a440174e3fc34439b319063e963f916a in term 1.
I20260620 12:25:26.670460 7659 leader_election.cc:302] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a440174e3fc34439b319063e963f916a, fe7a58f517d04d10aa1f256aa931244e; no voters:
I20260620 12:25:26.670624 7810 raft_consensus.cc:2781] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:26.670855 7810 raft_consensus.cc:686] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 1 LEADER]: Becoming Leader. State: Replica: a440174e3fc34439b319063e963f916a, State: Running, Role: LEADER
I20260620 12:25:26.671212 7810 consensus_queue.cc:237] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } }
I20260620 12:25:26.675016 6524 catalog_manager.cc:5495] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a reported cstate change: term changed from 0 to 1, leader changed from <none> to a440174e3fc34439b319063e963f916a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a440174e3fc34439b319063e963f916a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260620 12:25:26.754074 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.750752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.0.98.3:46617)
0620 12:25:26.750908 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.750912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.750926 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:26.751008 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.751010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.751072 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.751169 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:26.751414 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.751915 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.752703 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.752909 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.752985 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.753237 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:26.753723 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:26.753890 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.753918 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.753967 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:25:26.760212 7585 raft_consensus.cc:1261] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Refusing update from remote peer a440174e3fc34439b319063e963f916a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:26.760212 7448 raft_consensus.cc:1261] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 1 FOLLOWER]: Refusing update from remote peer a440174e3fc34439b319063e963f916a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:26.760973 7810 consensus_queue.cc:1035] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:26.761166 7813 consensus_queue.cc:1035] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:26.769428 7816 mvcc.cc:205] Tried to move back new op lower bound from 7298901306401812480 to 7298901306047713280. Current Snapshot: MvccSnapshot[applied={T|T < 7298901306401812480}]
I20260620 12:25:26.772089 7818 mvcc.cc:205] Tried to move back new op lower bound from 7298901306401812480 to 7298901306047713280. Current Snapshot: MvccSnapshot[applied={T|T < 7298901306401812480}]
I20260620 12:25:26.774587 7817 mvcc.cc:205] Tried to move back new op lower bound from 7298901306401812480 to 7298901306047713280. Current Snapshot: MvccSnapshot[applied={T|T < 7298901306401812480}]
I20260620 12:25:26.780241 6523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:26.780417 6523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:26.785389 6523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c1403069df443ce8ec099e16da75a42: Sending DeleteTablet for 3 replicas of tablet 914f118a944143c5ae1eb4ed2e5a02f9
I20260620 12:25:26.786142 7565 tablet_service.cc:1514] Processing DeleteTablet for tablet 914f118a944143c5ae1eb4ed2e5a02f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41364
I20260620 12:25:26.786170 7703 tablet_service.cc:1514] Processing DeleteTablet for tablet 914f118a944143c5ae1eb4ed2e5a02f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43286
I20260620 12:25:26.786907 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet 914f118a944143c5ae1eb4ed2e5a02f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33342
I20260620 12:25:26.787465 7827 tablet_replica.cc:330] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: stopping tablet replica
I20260620 12:25:26.787880 7827 raft_consensus.cc:2229] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:26.788340 7827 raft_consensus.cc:2258] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:26.788538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6492
I20260620 12:25:26.788792 7826 tablet_replica.cc:330] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: stopping tablet replica
I20260620 12:25:26.789106 7826 raft_consensus.cc:2229] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:26.789532 7826 raft_consensus.cc:2258] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:26.786623 7825 tablet_replica.cc:330] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: stopping tablet replica
I20260620 12:25:26.790200 7825 raft_consensus.cc:2229] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:26.790536 7825 raft_consensus.cc:2258] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:26.790637 7827 ts_tablet_manager.cc:1905] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:26.792107 7825 ts_tablet_manager.cc:1905] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:26.792928 7826 ts_tablet_manager.cc:1905] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:26.794787 7827 ts_tablet_manager.cc:1918] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:26.794941 7827 log.cc:1196] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/914f118a944143c5ae1eb4ed2e5a02f9
I20260620 12:25:26.795347 7827 ts_tablet_manager.cc:1939] T 914f118a944143c5ae1eb4ed2e5a02f9 P f245e2fae6a64ed49ca6086fa18cb731: Deleting consensus metadata
I20260620 12:25:26.795938 7825 ts_tablet_manager.cc:1918] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:26.796043 7825 log.cc:1196] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/914f118a944143c5ae1eb4ed2e5a02f9
I20260620 12:25:26.796427 7825 ts_tablet_manager.cc:1939] T 914f118a944143c5ae1eb4ed2e5a02f9 P fe7a58f517d04d10aa1f256aa931244e: Deleting consensus metadata
W20260620 12:25:26.799455 7518 connection.cc:535] server connection from 127.0.0.1:41364 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 12:25:26.799445 7383 connection.cc:535] server connection from 127.0.0.1:33342 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:25:26.800076 7826 ts_tablet_manager.cc:1918] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:26.800182 7826 log.cc:1196] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/914f118a944143c5ae1eb4ed2e5a02f9
I20260620 12:25:26.800542 7826 ts_tablet_manager.cc:1939] T 914f118a944143c5ae1eb4ed2e5a02f9 P a440174e3fc34439b319063e963f916a: Deleting consensus metadata
W20260620 12:25:26.801611 7657 connection.cc:441] server connection from 127.0.0.1:43286 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43286 (request call id 1) could send its response
I20260620 12:25:26.808419 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.787799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.98.61:33011)
0620 12:25:26.787998 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.788006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.793968 (+ 5962us) server_negotiation.cc:407] Connection header received
0620 12:25:26.794151 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.794154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.794210 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.794297 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.797878 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.798817 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.800687 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.801053 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.801156 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.803796 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.803807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.803811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.803844 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.805788 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.806420 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.806424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.806426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.806515 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.806863 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.806867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.806869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.807111 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.807242 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.808084 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.808231 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 12:25:26.815793 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.804723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.0.98.61:33011)
0620 12:25:26.805131 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.805136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.805672 (+ 536us) server_negotiation.cc:407] Connection header received
0620 12:25:26.805897 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.805900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.805946 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.806006 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.807123 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.807913 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.809259 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.809604 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.809675 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.811682 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.811686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.811688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.811714 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.813476 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.814257 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.814261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.814262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.814349 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.814749 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.814755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.814757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.814986 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.815095 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.815479 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.815605 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260620 12:25:26.816888 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.805099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.0.98.60:44753)
0620 12:25:26.805226 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.805233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.805651 (+ 418us) server_negotiation.cc:407] Connection header received
0620 12:25:26.805852 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.805856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.805914 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.805968 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.807133 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.807913 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.810569 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.810961 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.811052 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.813038 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.813042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.813045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.813073 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.814797 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.815516 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.815519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.815521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.815604 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.815910 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.815915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.815918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.816136 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.816261 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.816638 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.816754 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:25:26.818514 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:26.788250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.0.98.60:44753)
0620 12:25:26.788362 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:26.788366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:26.807998 (+ 19632us) server_negotiation.cc:407] Connection header received
0620 12:25:26.808032 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:26.808037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:26.808094 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:26.808168 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:26.809526 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.810793 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.812319 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:26.812721 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:26.812798 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:26.814736 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:26.814740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:26.814744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:26.814771 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:26.816599 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.817124 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.817128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.817130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.817219 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:26.817596 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:26.817600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:26.817602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:26.817806 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:26.817905 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:26.818300 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 12:25:26.818409 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":768}
I20260620 12:25:27.772668 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:27.772124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:41379)
0620 12:25:27.772421 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:27.772549 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":119,"threads_started":1}
W20260620 12:25:27.773051 7770 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:27.773919 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:27.773541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:37549)
0620 12:25:27.773757 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:27.773840 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
W20260620 12:25:27.774394 7632 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:27.776520 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:27.776168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:45031)
0620 12:25:27.776369 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:27.776446 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
W20260620 12:25:27.776889 7495 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:28.285537 7839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c1403069df443ce8ec099e16da75a42)
I20260620 12:25:28.285562 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c1403069df443ce8ec099e16da75a42)
I20260620 12:25:28.285799 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } }
I20260620 12:25:28.285838 7839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } }
I20260620 12:25:28.286960 7839 leader_election.cc:288] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c1403069df443ce8ec099e16da75a42 (127.0.98.62:45973), 155e4e6405e544bba95e8774e9cbccb4 (127.0.98.61:33011)
I20260620 12:25:28.287587 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.287094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46230)
0620 12:25:28.287401 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.287483 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":60,"threads_started":1}
I20260620 12:25:28.287714 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c1403069df443ce8ec099e16da75a42 (127.0.98.62:45973), 92e4976b2677431ab33a7af4d2b0562b (127.0.98.60:44753)
W20260620 12:25:28.287757 7301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:28.287954 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.287102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46236)
0620 12:25:28.287518 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.287602 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":77,"threads_started":1}
W20260620 12:25:28.288132 7112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:28.288982 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.288720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46252)
0620 12:25:28.288841 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.288897 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 12:25:28.289096 7301 leader_election.cc:334] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c1403069df443ce8ec099e16da75a42 (127.0.98.62:45973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:28.289126 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.288902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46254)
0620 12:25:28.289006 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.289058 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 12:25:28.289333 7112 leader_election.cc:334] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c1403069df443ce8ec099e16da75a42 (127.0.98.62:45973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:28.292315 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.287371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.0.98.61:33011)
0620 12:25:28.287825 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.287829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.288298 (+ 469us) server_negotiation.cc:407] Connection header received
0620 12:25:28.288447 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.288450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.288492 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.288548 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:28.288916 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.289803 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.290999 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.291796 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.291929 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.292031 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.292091 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.292166 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":330,"server-negotiator.queue_time_us":391,"thread_start_us":62,"threads_started":1}
I20260620 12:25:28.292265 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.287556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33011 (local address 127.0.0.1:57768)
0620 12:25:28.288183 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.288197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:28.288282 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:28.288687 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:28.288690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:28.288709 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:28.288822 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.288828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.289923 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.289926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.290874 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.290883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.291917 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.291920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.291990 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.292010 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:25:28.292059 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 12:25:28.292110 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":73,"threads_started":1}
I20260620 12:25:28.292769 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92e4976b2677431ab33a7af4d2b0562b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "155e4e6405e544bba95e8774e9cbccb4" is_pre_election: true
I20260620 12:25:28.293006 7136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e4976b2677431ab33a7af4d2b0562b in term 1.
I20260620 12:25:28.293504 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.287553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.0.98.60:44753)
0620 12:25:28.287958 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.287965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.288130 (+ 165us) server_negotiation.cc:407] Connection header received
0620 12:25:28.288311 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.288315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.288389 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.289291 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:28.289402 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.290262 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.292277 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.292896 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.292996 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.293058 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.293371 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.293427 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":83,"threads_started":1}
I20260620 12:25:28.293502 7302 leader_election.cc:302] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155e4e6405e544bba95e8774e9cbccb4, 92e4976b2677431ab33a7af4d2b0562b; no voters: 0c1403069df443ce8ec099e16da75a42
I20260620 12:25:28.293612 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.287151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:58360)
0620 12:25:28.288009 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.288025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:28.288139 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:28.288558 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:28.288565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:28.288588 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:28.288730 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.288739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.290380 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.290387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.291798 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.291806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.293027 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.293030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.293176 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.293209 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 12:25:28.293355 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 12:25:28.293439 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":38,"thread_start_us":57,"threads_started":1}
I20260620 12:25:28.294003 7839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 12:25:28.294077 7839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c1403069df443ce8ec099e16da75a42)
I20260620 12:25:28.294137 7839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:25:28.294145 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "155e4e6405e544bba95e8774e9cbccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92e4976b2677431ab33a7af4d2b0562b" is_pre_election: true
I20260620 12:25:28.295583 7839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } }
I20260620 12:25:28.295853 7335 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 155e4e6405e544bba95e8774e9cbccb4 in current term 2: Already voted for candidate 92e4976b2677431ab33a7af4d2b0562b in this term.
I20260620 12:25:28.296047 7839 leader_election.cc:288] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [CANDIDATE]: Term 2 election: Requested vote from peers 0c1403069df443ce8ec099e16da75a42 (127.0.98.62:45973), 155e4e6405e544bba95e8774e9cbccb4 (127.0.98.61:33011)
I20260620 12:25:28.296458 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.296134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46268)
0620 12:25:28.296288 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.296384 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 12:25:28.296452 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92e4976b2677431ab33a7af4d2b0562b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "155e4e6405e544bba95e8774e9cbccb4"
I20260620 12:25:28.296608 7136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:25:28.296702 7110 leader_election.cc:302] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 155e4e6405e544bba95e8774e9cbccb4; no voters: 0c1403069df443ce8ec099e16da75a42, 92e4976b2677431ab33a7af4d2b0562b
I20260620 12:25:28.297106 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.296937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46270)
0620 12:25:28.297010 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.297056 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 12:25:28.297230 7301 leader_election.cc:334] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0c1403069df443ce8ec099e16da75a42 (127.0.98.62:45973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
I20260620 12:25:28.297890 7136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e4976b2677431ab33a7af4d2b0562b in term 2.
I20260620 12:25:28.298015 7840 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:25:28.298203 7302 leader_election.cc:302] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155e4e6405e544bba95e8774e9cbccb4, 92e4976b2677431ab33a7af4d2b0562b; no voters: 0c1403069df443ce8ec099e16da75a42
I20260620 12:25:28.298411 7839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:25:28.298614 7839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [term 2 LEADER]: Becoming Leader. State: Replica: 92e4976b2677431ab33a7af4d2b0562b, State: Running, Role: LEADER
I20260620 12:25:28.298885 7839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } }
I20260620 12:25:28.300011 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e4976b2677431ab33a7af4d2b0562b. Latest consensus state: current_term: 2 leader_uuid: "92e4976b2677431ab33a7af4d2b0562b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:28.300099 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:28.300392 7851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:28.302448 7851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:28.302894 7851 catalog_manager.cc:1259] Loaded cluster ID: 4d6161b194794dc484afd4156351d26c
I20260620 12:25:28.302939 7851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:28.303484 7851 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:28.303874 7851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Loaded TSK: 0
I20260620 12:25:28.304489 7851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:28.333323 7324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:28.336994 7324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:28.339921 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.339658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46284)
0620 12:25:28.339774 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.339847 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 12:25:28.340054 7136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [term 2 FOLLOWER]: Refusing update from remote peer 92e4976b2677431ab33a7af4d2b0562b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 12:25:28.340703 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.340445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.0.1:46286)
0620 12:25:28.340569 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.340631 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 12:25:28.340754 7849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [LEADER]: Connected to new peer: Peer: permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 12:25:28.340935 7301 consensus_peers.cc:487] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b -> Peer 0c1403069df443ce8ec099e16da75a42 (127.0.98.62:45973): Couldn't send request to peer 0c1403069df443ce8ec099e16da75a42. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 12:25:28.342374 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e4976b2677431ab33a7af4d2b0562b. Latest consensus state: current_term: 2 leader_uuid: "92e4976b2677431ab33a7af4d2b0562b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:28.342532 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:25:28.343736 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "92e4976b2677431ab33a7af4d2b0562b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:28.343863 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:28.343981 7853 mvcc.cc:205] Tried to move back new op lower bound from 7298901312875655168 to 7298901312714977280. Current Snapshot: MvccSnapshot[applied={T|T < 7298901312875655168}]
I20260620 12:25:28.344137 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "92e4976b2677431ab33a7af4d2b0562b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c1403069df443ce8ec099e16da75a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45973 } } peers { permanent_uuid: "155e4e6405e544bba95e8774e9cbccb4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33011 } attrs { promote: false } } peers { permanent_uuid: "92e4976b2677431ab33a7af4d2b0562b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attrs { promote: false } } }
I20260620 12:25:28.344238 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155e4e6405e544bba95e8774e9cbccb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:25:28.356819 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.350699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46617 (local address 127.0.0.1:39268)
0620 12:25:28.350948 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.350961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:28.351045 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:28.352487 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:28.352490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:28.352506 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:28.352764 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.352774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.354092 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.354097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.355382 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.355394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.356386 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.356389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.356510 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.356533 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:25:28.356604 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 12:25:28.356672 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":70}
I20260620 12:25:28.357553 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.351203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.0.98.3:46617)
0620 12:25:28.351535 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.351542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.351558 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:25:28.351635 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.351638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.351695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.351791 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:28.352898 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.353708 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.355520 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.356250 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.356378 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.356445 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.357334 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.357428 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260620 12:25:28.357600 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.351203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39413 (local address 127.0.0.1:42524)
0620 12:25:28.352009 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.352026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:28.352127 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:28.352572 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:28.352574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:28.352589 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:28.352723 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.352732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.353825 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.353830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.355291 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.355300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.357232 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.357236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.357334 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.357356 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:25:28.357421 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:25:28.357482 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":30,"thread_start_us":85,"threads_started":1}
I20260620 12:25:28.358475 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.350878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.0.98.2:39413)
0620 12:25:28.351656 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.351660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.352230 (+ 570us) server_negotiation.cc:407] Connection header received
0620 12:25:28.352292 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.352297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.352367 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.352455 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:28.352860 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.353702 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.355433 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.357092 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.357219 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.357279 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.358106 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.358280 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":164,"threads_started":1}
I20260620 12:25:28.358806 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.350219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39003 (local address 127.0.0.1:52004)
0620 12:25:28.350457 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.350472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:28.350551 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:28.352107 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:28.352110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:28.352128 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:28.352229 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.352234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.355673 (+ 3439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.355678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.356872 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.356882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.358340 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.358344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.358464 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.358495 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:25:28.358547 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 12:25:28.358629 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 12:25:28.359067 7703 tablet_service.cc:1467] Processing CreateTablet for tablet b26f56dc6c4d404fa425f842aff8be7d (DEFAULT_TABLE table=test-table [id=4af601342f79411c8ce0a0e4c577c8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:28.359387 7703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26f56dc6c4d404fa425f842aff8be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:28.359515 7565 tablet_service.cc:1467] Processing CreateTablet for tablet b26f56dc6c4d404fa425f842aff8be7d (DEFAULT_TABLE table=test-table [id=4af601342f79411c8ce0a0e4c577c8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:28.359618 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.350379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.98.1:39003)
0620 12:25:28.351864 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.351870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.351897 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:25:28.351955 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.351958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.351997 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.352098 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:28.354824 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.355566 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.357217 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.358196 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.359300 (+ 1104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.359364 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.359413 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.359480 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":721,"threads_started":1}
I20260620 12:25:28.359782 7565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26f56dc6c4d404fa425f842aff8be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:28.361883 7428 tablet_service.cc:1467] Processing CreateTablet for tablet b26f56dc6c4d404fa425f842aff8be7d (DEFAULT_TABLE table=test-table [id=4af601342f79411c8ce0a0e4c577c8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:28.362143 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26f56dc6c4d404fa425f842aff8be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:28.362437 7863 tablet_bootstrap.cc:492] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: Bootstrap starting.
I20260620 12:25:28.363507 7863 tablet_bootstrap.cc:654] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:28.364522 7863 tablet_bootstrap.cc:492] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: No bootstrap required, opened a new log
I20260620 12:25:28.364640 7863 ts_tablet_manager.cc:1397] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:28.364653 7865 tablet_bootstrap.cc:492] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731: Bootstrap starting.
I20260620 12:25:28.365159 7862 tablet_bootstrap.cc:492] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: Bootstrap starting.
I20260620 12:25:28.365111 7863 raft_consensus.cc:348] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.365249 7863 raft_consensus.cc:374] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:28.365290 7863 raft_consensus.cc:729] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7a58f517d04d10aa1f256aa931244e, State: Initialized, Role: FOLLOWER
I20260620 12:25:28.365442 7863 consensus_queue.cc:260] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.365770 7863 ts_tablet_manager.cc:1428] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:28.365882 7865 tablet_bootstrap.cc:654] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:28.366029 7862 tablet_bootstrap.cc:654] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:28.366979 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.366554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:47347)
0620 12:25:28.366817 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.366877 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
I20260620 12:25:28.367139 7862 tablet_bootstrap.cc:492] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: No bootstrap required, opened a new log
I20260620 12:25:28.367221 7862 ts_tablet_manager.cc:1397] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 12:25:28.367632 7862 raft_consensus.cc:348] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.367723 7862 raft_consensus.cc:374] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:28.367745 7862 raft_consensus.cc:729] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a440174e3fc34439b319063e963f916a, State: Initialized, Role: FOLLOWER
I20260620 12:25:28.367887 7862 consensus_queue.cc:260] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.368229 7862 ts_tablet_manager.cc:1428] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:28.368553 7631 heartbeater.cc:503] Master 127.0.98.60:44753 was elected leader, sending a full tablet report...
I20260620 12:25:28.368920 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.368475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:55571)
0620 12:25:28.368738 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.368813 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260620 12:25:28.369545 7865 tablet_bootstrap.cc:492] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731: No bootstrap required, opened a new log
I20260620 12:25:28.369657 7865 ts_tablet_manager.cc:1397] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 12:25:28.369910 7769 heartbeater.cc:503] Master 127.0.98.60:44753 was elected leader, sending a full tablet report...
I20260620 12:25:28.370114 7865 raft_consensus.cc:348] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.370232 7865 raft_consensus.cc:374] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:28.370285 7865 raft_consensus.cc:729] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f245e2fae6a64ed49ca6086fa18cb731, State: Initialized, Role: FOLLOWER
I20260620 12:25:28.370426 7865 consensus_queue.cc:260] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.370779 7865 ts_tablet_manager.cc:1428] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:25:28.371388 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.371148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:51647)
0620 12:25:28.371249 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.371306 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 12:25:28.372424 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.371404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:41155)
0620 12:25:28.372255 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.372322 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":83,"threads_started":1}
I20260620 12:25:28.372994 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.372754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:53443)
0620 12:25:28.372904 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.372943 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 12:25:28.447188 7872 raft_consensus.cc:482] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:28.447345 7872 raft_consensus.cc:504] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.448589 7872 leader_election.cc:288] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a440174e3fc34439b319063e963f916a (127.0.98.3:46617), fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413)
I20260620 12:25:28.453043 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.449057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38411 (local address 127.0.98.3:46617)
0620 12:25:28.449175 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.449180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.449507 (+ 327us) server_negotiation.cc:407] Connection header received
0620 12:25:28.449643 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.449647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.449704 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.449751 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:28.450165 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.450908 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.451992 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.452406 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.452532 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.452600 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.452840 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.452915 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:25:28.453056 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.449082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39613 (local address 127.0.98.2:39413)
0620 12:25:28.449210 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.449214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.449232 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:25:28.449423 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.449428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.449500 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.449578 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:28.449975 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.450814 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.452011 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.452448 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.452568 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.452634 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.452852 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.452932 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:28.453056 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.448878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39413 (local address 127.0.98.1:39613)
0620 12:25:28.449160 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.449176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:28.449285 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:28.449698 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:28.449701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:28.449717 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:28.449855 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.449866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.450945 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.450949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.451877 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.451887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.452572 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.452575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.452728 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.452745 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:25:28.452835 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:25:28.452912 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":82}
I20260620 12:25:28.453076 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.448878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46617 (local address 127.0.98.1:38411)
0620 12:25:28.449392 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.449405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:28.449509 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:28.449912 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:28.449917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:28.449938 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:28.450050 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.450055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.451002 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.451007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.451862 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:28.451871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.452549 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.452553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:28.452697 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.452721 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:25:28.452813 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 12:25:28.452895 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":140,"thread_start_us":134,"threads_started":1}
I20260620 12:25:28.453533 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26f56dc6c4d404fa425f842aff8be7d" candidate_uuid: "f245e2fae6a64ed49ca6086fa18cb731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a58f517d04d10aa1f256aa931244e" is_pre_election: true
I20260620 12:25:28.453718 7585 raft_consensus.cc:2454] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f245e2fae6a64ed49ca6086fa18cb731 in term 0.
I20260620 12:25:28.453801 7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26f56dc6c4d404fa425f842aff8be7d" candidate_uuid: "f245e2fae6a64ed49ca6086fa18cb731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a440174e3fc34439b319063e963f916a" is_pre_election: true
I20260620 12:25:28.454053 7723 raft_consensus.cc:2454] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f245e2fae6a64ed49ca6086fa18cb731 in term 0.
I20260620 12:25:28.454092 7384 leader_election.cc:302] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f245e2fae6a64ed49ca6086fa18cb731, fe7a58f517d04d10aa1f256aa931244e; no voters:
I20260620 12:25:28.454391 7872 raft_consensus.cc:2781] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:28.454455 7872 raft_consensus.cc:482] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:28.454491 7872 raft_consensus.cc:3037] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:28.455605 7872 raft_consensus.cc:504] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.455957 7872 leader_election.cc:288] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [CANDIDATE]: Term 1 election: Requested vote from peers a440174e3fc34439b319063e963f916a (127.0.98.3:46617), fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413)
I20260620 12:25:28.456282 7723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26f56dc6c4d404fa425f842aff8be7d" candidate_uuid: "f245e2fae6a64ed49ca6086fa18cb731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a440174e3fc34439b319063e963f916a"
I20260620 12:25:28.456406 7723 raft_consensus.cc:3037] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:28.456353 7585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26f56dc6c4d404fa425f842aff8be7d" candidate_uuid: "f245e2fae6a64ed49ca6086fa18cb731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a58f517d04d10aa1f256aa931244e"
I20260620 12:25:28.456506 7585 raft_consensus.cc:3037] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:28.457355 7723 raft_consensus.cc:2454] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f245e2fae6a64ed49ca6086fa18cb731 in term 1.
I20260620 12:25:28.457425 7585 raft_consensus.cc:2454] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f245e2fae6a64ed49ca6086fa18cb731 in term 1.
I20260620 12:25:28.457646 7382 leader_election.cc:302] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a440174e3fc34439b319063e963f916a, f245e2fae6a64ed49ca6086fa18cb731; no voters:
I20260620 12:25:28.457826 7872 raft_consensus.cc:2781] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:28.458014 7872 raft_consensus.cc:686] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 1 LEADER]: Becoming Leader. State: Replica: f245e2fae6a64ed49ca6086fa18cb731, State: Running, Role: LEADER
I20260620 12:25:28.458227 7494 heartbeater.cc:503] Master 127.0.98.60:44753 was elected leader, sending a full tablet report...
I20260620 12:25:28.458307 7872 consensus_queue.cc:237] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } }
I20260620 12:25:28.458793 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.458457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:34701)
0620 12:25:28.458626 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.458716 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 12:25:28.460848 7325 catalog_manager.cc:5495] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 reported cstate change: term changed from 0 to 1, leader changed from <none> to f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f245e2fae6a64ed49ca6086fa18cb731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f245e2fae6a64ed49ca6086fa18cb731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 } health_report { overall_health: UNKNOWN } } }
I20260620 12:25:28.481818 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.477992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.0.98.1:39003)
0620 12:25:28.478123 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:28.478126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:28.478139 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:25:28.478253 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:28.478255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:28.478309 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:28.478403 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:28.478725 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.479229 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.480403 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:28.480620 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:28.480702 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:28.480957 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:28.481415 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:28.481503 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:28.481617 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 12:25:28.481661 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:25:28.484576 7585 raft_consensus.cc:1261] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Refusing update from remote peer f245e2fae6a64ed49ca6086fa18cb731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:28.484833 7723 raft_consensus.cc:1261] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 1 FOLLOWER]: Refusing update from remote peer f245e2fae6a64ed49ca6086fa18cb731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:28.485105 7875 consensus_queue.cc:1035] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7a58f517d04d10aa1f256aa931244e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:28.485317 7872 consensus_queue.cc:1035] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a440174e3fc34439b319063e963f916a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:28.486864 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.486595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:46371)
0620 12:25:28.486733 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.486789 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 12:25:28.487198 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.486925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:53271)
0620 12:25:28.487063 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.487121 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 12:25:28.489830 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.489577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:46743)
0620 12:25:28.489710 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.489757 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 12:25:28.490976 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.490735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.3:38841)
0620 12:25:28.490870 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.490916 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 12:25:28.494680 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.494415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.2:56397)
0620 12:25:28.494545 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.494591 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 12:25:28.495890 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.495621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45973 (local address 127.0.98.1:44149)
0620 12:25:28.495753 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.495818 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 12:25:28.498616 7325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:28.498795 7325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:28.502616 7325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92e4976b2677431ab33a7af4d2b0562b: Sending DeleteTablet for 3 replicas of tablet b26f56dc6c4d404fa425f842aff8be7d
I20260620 12:25:28.503373 7703 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f56dc6c4d404fa425f842aff8be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:28 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39268
I20260620 12:25:28.503399 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f56dc6c4d404fa425f842aff8be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:28 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52004
I20260620 12:25:28.503484 7565 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f56dc6c4d404fa425f842aff8be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:28 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42524
I20260620 12:25:28.503743 7889 tablet_replica.cc:330] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731: stopping tablet replica
I20260620 12:25:28.503878 7889 raft_consensus.cc:2229] T b26f56dc6c4d404fa425f842aff8be7d P f245e2fae6a64ed49ca6086fa18cb731 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:28.504081 7888 tablet_replica.cc:330] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: stopping tablet replica
I20260620 12:25:28.504205 7888 raft_consensus.cc:2229] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:28.504351 7888 raft_consensus.cc:2258] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:28.504469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 7270
I20260620 12:25:28.506042 7888 ts_tablet_manager.cc:1905] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:28.506841 7890 tablet_replica.cc:330] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: stopping tablet replica
I20260620 12:25:28.506978 7890 raft_consensus.cc:2229] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:28.507112 7890 raft_consensus.cc:2258] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:28.507717 7890 ts_tablet_manager.cc:1905] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:28.509306 7888 ts_tablet_manager.cc:1918] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:28.509389 7888 log.cc:1196] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/b26f56dc6c4d404fa425f842aff8be7d
I20260620 12:25:28.509684 7888 ts_tablet_manager.cc:1939] T b26f56dc6c4d404fa425f842aff8be7d P a440174e3fc34439b319063e963f916a: Deleting consensus metadata
I20260620 12:25:28.509752 7890 ts_tablet_manager.cc:1918] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:28.509833 7890 log.cc:1196] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/b26f56dc6c4d404fa425f842aff8be7d
I20260620 12:25:28.510128 7890 ts_tablet_manager.cc:1939] T b26f56dc6c4d404fa425f842aff8be7d P fe7a58f517d04d10aa1f256aa931244e: Deleting consensus metadata
I20260620 12:25:28.510555 7300 catalog_manager.cc:4841] TS a440174e3fc34439b319063e963f916a (127.0.98.3:46617): tablet b26f56dc6c4d404fa425f842aff8be7d (table test-table [id=4af601342f79411c8ce0a0e4c577c8ca]) successfully deleted
I20260620 12:25:28.510789 7302 catalog_manager.cc:4841] TS fe7a58f517d04d10aa1f256aa931244e (127.0.98.2:39413): tablet b26f56dc6c4d404fa425f842aff8be7d (table test-table [id=4af601342f79411c8ce0a0e4c577c8ca]) successfully deleted
I20260620 12:25:28.512387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 7502
I20260620 12:25:28.512827 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:28.512445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39003 (local address 127.0.0.1:52022)
0620 12:25:28.512669 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:28.512729 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260620 12:25:28.513010 7302 catalog_manager.cc:4584] TS f245e2fae6a64ed49ca6086fa18cb731 (127.0.98.1:39003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b26f56dc6c4d404fa425f842aff8be7d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39003: connect: Connection refused (error 111)
I20260620 12:25:28.518455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 7638
I20260620 12:25:28.526275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 6988
I20260620 12:25:28.532886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 7090
2026-06-20T12:25:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5573 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:28 dist-test-slave-bwpp krb5kdc[7893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:28 dist-test-slave-bwpp krb5kdc[7893](info): set up 2 sockets
Jun 20 12:25:28 dist-test-slave-bwpp krb5kdc[7893](info): commencing operation
krb5kdc: starting...
W20260620 12:25:29.579296 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:29 dist-test-slave-bwpp krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 12:25:29.701229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44045
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:29.815071 7906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:29.815361 7906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:29.815405 7906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:29.819335 7906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:29.819393 7906 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:25:29.819414 7906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:29.819435 7906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:29.819454 7906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:29.819479 7906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:29.824899 7906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44045
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:29.825951 7906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:29.827072 7906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:29.835701 7912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:29.835752 7914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:29.835752 7911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:29.835917 7906 server_base.cc:1034] running on GCE node
I20260620 12:25:29.836534 7906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:29.837730 7906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:29.838897 7906 hybrid_clock.cc:648] HybridClock initialized: now 1781958329838873 us; error 34 us; skew 500 ppm
Jun 20 12:25:29 dist-test-slave-bwpp krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:29.841868 7906 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 12:25:29.842944 7906 webserver.cc:460] Webserver started at http://127.0.98.62:40815/ using document root <none> and password file <none>
I20260620 12:25:29.843523 7906 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:29.843590 7906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:29.843853 7906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:29.846194 7906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "9a668d61a111464d93fe9e6f2af23e20"
format_stamp: "Formatted at 2026-06-20 12:25:29 on dist-test-slave-bwpp"
server_key: "9e2be2c4c6fb5a6c355152deed590684"
server_key_iv: "00b8889031d117a5c2e04634e8103e89"
server_key_version: "encryptionkey@0"
I20260620 12:25:29.846846 7906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a668d61a111464d93fe9e6f2af23e20"
format_stamp: "Formatted at 2026-06-20 12:25:29 on dist-test-slave-bwpp"
server_key: "9e2be2c4c6fb5a6c355152deed590684"
server_key_iv: "00b8889031d117a5c2e04634e8103e89"
server_key_version: "encryptionkey@0"
I20260620 12:25:29.850862 7906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:25:29.853338 7921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:29.854638 7906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:29.854849 7906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "9a668d61a111464d93fe9e6f2af23e20"
format_stamp: "Formatted at 2026-06-20 12:25:29 on dist-test-slave-bwpp"
server_key: "9e2be2c4c6fb5a6c355152deed590684"
server_key_iv: "00b8889031d117a5c2e04634e8103e89"
server_key_version: "encryptionkey@0"
I20260620 12:25:29.854969 7906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:29.875314 7906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:29.878938 7906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:29.879168 7906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:29.887492 7906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44045
I20260620 12:25:29.887522 7983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44045 every 8 connection(s)
I20260620 12:25:29.888469 7906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:29.891362 7984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:29.896912 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Bootstrap starting.
I20260620 12:25:29.897781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 7906
I20260620 12:25:29.897888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:29.898196 392 external_mini_cluster.cc:1428] Setting key b401c8eeecd170461f7b78f4c7732cae
I20260620 12:25:29.899823 7984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:29.900589 7984 log.cc:824] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:29.902549 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: No bootstrap required, opened a new log
I20260620 12:25:29.905946 7984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a668d61a111464d93fe9e6f2af23e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44045 } }
I20260620 12:25:29.906193 7984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:29.906275 7984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a668d61a111464d93fe9e6f2af23e20, State: Initialized, Role: FOLLOWER
I20260620 12:25:29.906833 7984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a668d61a111464d93fe9e6f2af23e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44045 } }
I20260620 12:25:29.907018 7984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:29.907084 7984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:29.907186 7984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 12:25:29 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:29.908156 7984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a668d61a111464d93fe9e6f2af23e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44045 } }
I20260620 12:25:29.908535 7984 leader_election.cc:302] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a668d61a111464d93fe9e6f2af23e20; no voters:
I20260620 12:25:29.908969 7984 leader_election.cc:288] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:29.909049 7989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:29.909485 7989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [term 1 LEADER]: Becoming Leader. State: Replica: 9a668d61a111464d93fe9e6f2af23e20, State: Running, Role: LEADER
I20260620 12:25:29.909789 7989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a668d61a111464d93fe9e6f2af23e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44045 } }
I20260620 12:25:29.910336 7984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:29.911064 7991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a668d61a111464d93fe9e6f2af23e20. Latest consensus state: current_term: 1 leader_uuid: "9a668d61a111464d93fe9e6f2af23e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a668d61a111464d93fe9e6f2af23e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44045 } } }
I20260620 12:25:29.911206 7991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:29.911604 7997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:29.911710 7990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a668d61a111464d93fe9e6f2af23e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a668d61a111464d93fe9e6f2af23e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44045 } } }
I20260620 12:25:29.911826 7990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:29.914364 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:29.899423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.0.98.62:44045)
0620 12:25:29.899891 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:29.899906 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:29.899966 (+ 60us) server_negotiation.cc:407] Connection header received
0620 12:25:29.900634 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:29.900652 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:29.901108 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:29.901533 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:29.902386 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:29.903478 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:29.904393 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:29.904908 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:29.905092 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:29.908118 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:29.908125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:29.908149 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:29.908177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:29.910353 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:29.910973 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:29.910978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:29.910985 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:29.911136 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:29.911722 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:29.911729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:29.911761 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:29.912473 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:29.912703 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:29.912958 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 12:25:29.913306 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":145,"threads_started":1}
I20260620 12:25:29.914919 7997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:29.920951 7997 catalog_manager.cc:1347] Generated new cluster ID: e397b62e6ddb4ccaa4693cab3bcaf3a8
I20260620 12:25:29.921027 7997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:29.940490 7997 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:29.941541 7997 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:29.949654 7997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Generated new TSK 0
I20260620 12:25:29.950398 7997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 12:25:29.982463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44045
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:30.104667 8010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:30.104967 8010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:30.105042 8010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:30.109084 8010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:30.109166 8010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:30.109229 8010 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:25:30.109313 8010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:30.115471 8010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:30.116570 8010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:30.117616 8010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:30.127059 8018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:30.127076 8016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:30.127076 8015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:30.127151 8010 server_base.cc:1034] running on GCE node
I20260620 12:25:30.127739 8010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:30.128396 8010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:30.129580 8010 hybrid_clock.cc:648] HybridClock initialized: now 1781958330129566 us; error 26 us; skew 500 ppm
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:30.132328 8010 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:25:30.133198 8010 webserver.cc:460] Webserver started at http://127.0.98.1:34575/ using document root <none> and password file <none>
I20260620 12:25:30.133744 8010 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:30.133832 8010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:30.134061 8010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:30.136091 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "9aa2d938366645bfacbc42f40ba173fc"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "46ee5938afd8cb03effca2f09402f704"
server_key_iv: "761d3f32fb258ceb98349fbee0c00d57"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.136647 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9aa2d938366645bfacbc42f40ba173fc"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "46ee5938afd8cb03effca2f09402f704"
server_key_iv: "761d3f32fb258ceb98349fbee0c00d57"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.140339 8010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:30.143005 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:30.144816 8010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:30.144937 8010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "9aa2d938366645bfacbc42f40ba173fc"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "46ee5938afd8cb03effca2f09402f704"
server_key_iv: "761d3f32fb258ceb98349fbee0c00d57"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.145063 8010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:30.181736 8010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:30.185128 8010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:30.185334 8010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:30.186837 8010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:30.186908 8010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:30.186970 8010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:30.187026 8010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:30.198014 8010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36885
I20260620 12:25:30.198089 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36885 every 8 connection(s)
I20260620 12:25:30.199060 8010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:30.200035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 8010
I20260620 12:25:30.200160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:30.200516 392 external_mini_cluster.cc:1428] Setting key 6cc4731285f2e129c5d688dabe28dd2e
I20260620 12:25:30.202566 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.187714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57867 (local address 127.0.98.62:44045)
0620 12:25:30.187845 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.187849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.188903 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 12:25:30.189935 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.189940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.190023 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.190088 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.191925 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.192821 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.194595 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.195015 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.195133 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.198182 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.198187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.198190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.198217 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.199830 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.200431 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.200436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.200438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.200538 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.200894 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.200897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.200900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.201145 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.201268 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.202187 (+ 919us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.202322 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:25:30.203207 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.188268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44045 (local address 127.0.98.1:57867)
0620 12:25:30.188738 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.188780 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.189670 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.190543 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.190556 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.191083 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:30.191728 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.191751 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.192962 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.192966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.194475 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.194486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.195169 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.195173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.195356 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.195957 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:30.195978 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:30.197991 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:30.199960 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.199967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.199985 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.200329 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.200646 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.200649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.200651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.200786 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.201704 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:30.201710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:30.201958 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.202110 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.202353 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
I20260620 12:25:30.204461 8141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44045
I20260620 12:25:30.204632 8141 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:30.205199 8141 heartbeater.cc:511] Master 127.0.98.62:44045 requested a full tablet report, sending...
I20260620 12:25:30.206975 7937 ts_manager.cc:194] Registered new tserver with Master: 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885)
I20260620 12:25:30.208720 7937 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57867
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:30.214385 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.206716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43603 (local address 127.0.98.62:44045)
0620 12:25:30.206826 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.206830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.207040 (+ 210us) server_negotiation.cc:407] Connection header received
0620 12:25:30.207198 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.207201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.207248 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.207341 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.207739 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.208536 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.209405 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.209791 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.209915 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.211254 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.211259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.211262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.211285 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.212722 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.213163 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.213167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.213169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.213269 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.213566 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.213571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.213573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.213756 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.213869 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.214121 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.214229 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":260,"server-negotiator.queue_time_us":49}
I20260620 12:25:30.214391 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.206638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44045 (local address 127.0.98.1:43603)
0620 12:25:30.206958 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.206969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.207117 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.207351 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.207354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.207543 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:30.207631 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.207636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.208657 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.208661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.209300 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.209313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.209939 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.209943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.210024 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.210567 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:30.210578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:30.211078 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:30.212840 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.212843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.212846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.213080 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.213384 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.213386 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.213388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.213473 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.213859 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:30.213862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:30.213956 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.214062 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.214194 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 12:25:30.227747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44045
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:30.336133 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:30.336371 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:30.336434 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:30.340502 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:30.340556 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:30.340595 8150 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:25:30.340627 8150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:30.345726 8150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:30.346669 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:30.347617 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:30.356997 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:30.356983 8150 server_base.cc:1034] running on GCE node
W20260620 12:25:30.356966 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:30.356961 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:30.357544 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:30.358199 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:30.359383 8150 hybrid_clock.cc:648] HybridClock initialized: now 1781958330359353 us; error 43 us; skew 500 ppm
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:30.362099 8150 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:25:30.362967 8150 webserver.cc:460] Webserver started at http://127.0.98.2:37943/ using document root <none> and password file <none>
I20260620 12:25:30.363497 8150 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:30.363582 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:30.363802 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:30.365736 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "84bcb8c883b8496c8ea38ae6db6214dd"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "33db61afa3092832c4434aad83a06317"
server_key_iv: "4ac62e030f42c706e548f9ca552cb033"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.366284 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84bcb8c883b8496c8ea38ae6db6214dd"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "33db61afa3092832c4434aad83a06317"
server_key_iv: "4ac62e030f42c706e548f9ca552cb033"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.369760 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:25:30.372113 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:30.373289 8150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:30.373451 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "84bcb8c883b8496c8ea38ae6db6214dd"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "33db61afa3092832c4434aad83a06317"
server_key_iv: "4ac62e030f42c706e548f9ca552cb033"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.373570 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:30.383582 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:30.386889 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:30.387094 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:30.388490 8150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:30.388568 8150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:30.388648 8150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:30.388695 8150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:30.400398 8150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33063
I20260620 12:25:30.400475 8280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33063 every 8 connection(s)
I20260620 12:25:30.401559 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:30.403465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 8150
I20260620 12:25:30.403616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:30.403750 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.389281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55937 (local address 127.0.98.62:44045)
0620 12:25:30.389403 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.389406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.390422 (+ 1016us) server_negotiation.cc:407] Connection header received
0620 12:25:30.391554 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.391557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.391614 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.391709 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.393299 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.394126 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.396015 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.396384 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.396502 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.399577 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.399582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.399585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.399616 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.401502 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.402054 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.402057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.402059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.402148 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.402487 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.402497 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.402499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.402737 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.402848 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.403411 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.403516 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:25:30.403923 392 external_mini_cluster.cc:1428] Setting key 19f14b8589230218ee696087a98a493d
I20260620 12:25:30.404603 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.389810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44045 (local address 127.0.98.2:55937)
0620 12:25:30.390276 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.390314 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.391339 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.391928 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.391938 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.392451 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:30.393126 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.393148 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.394300 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.394304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.395856 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.395871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.396578 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.396584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.396812 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.397406 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:30.397435 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:30.399410 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:30.401646 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.401652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.401666 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.401918 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.402241 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.402246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.402248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.402383 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.402856 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:30.402864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:30.403097 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.403303 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.403641 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":142,"threads_started":1}
I20260620 12:25:30.405885 8281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44045
I20260620 12:25:30.406018 8281 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:30.406533 8281 heartbeater.cc:511] Master 127.0.98.62:44045 requested a full tablet report, sending...
I20260620 12:25:30.408092 7937 ts_manager.cc:194] Registered new tserver with Master: 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063)
I20260620 12:25:30.408663 7937 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55937
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:25:30.416344 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.408155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40687 (local address 127.0.98.62:44045)
0620 12:25:30.408323 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.408329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.409168 (+ 839us) server_negotiation.cc:407] Connection header received
0620 12:25:30.409266 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.409272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.409326 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.409415 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.409876 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.410686 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.411742 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.411981 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.412096 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.413312 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.413315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.413317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.413337 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.414800 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.415238 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.415244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.415249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.415332 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.415613 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.415615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.415616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.415795 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.415872 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.416148 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.416235 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:25:30.416451 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.408035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44045 (local address 127.0.98.2:40687)
0620 12:25:30.409073 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.409089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.409188 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.409446 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.409449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.409646 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:30.409749 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.409756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.410796 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.410799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.411618 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.411628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.412098 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.412101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.412196 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.412711 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:30.412729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:30.413176 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:30.414895 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.414898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.414899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.415132 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.415425 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.415428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.415430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.415522 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.415876 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:30.415879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:30.415986 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.416106 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.416229 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 12:25:30.431726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44045
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:30.542865 8290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:30.543159 8290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:30.543242 8290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:30.547236 8290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:30.547333 8290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:30.547406 8290 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:25:30.547502 8290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:30.552780 8290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:30.553830 8290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:30.554870 8290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:30.564617 8295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:30.564615 8298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:30.564638 8296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:30.564808 8290 server_base.cc:1034] running on GCE node
I20260620 12:25:30.565428 8290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:30.566052 8290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:30.567238 8290 hybrid_clock.cc:648] HybridClock initialized: now 1781958330567226 us; error 26 us; skew 500 ppm
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:30.569974 8290 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:25:30.570878 8290 webserver.cc:460] Webserver started at http://127.0.98.3:46575/ using document root <none> and password file <none>
I20260620 12:25:30.571403 8290 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:30.571461 8290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:30.571624 8290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:30.573598 8290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "a07753878eee4f728c49d6606998adf3"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "379ab48f28805c99032aa7959dee7024"
server_key_iv: "68bd5aad64a7fdc0fdfa998a227d4d7f"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.574117 8290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a07753878eee4f728c49d6606998adf3"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "379ab48f28805c99032aa7959dee7024"
server_key_iv: "68bd5aad64a7fdc0fdfa998a227d4d7f"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.577942 8290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:25:30.580421 8305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:30.581640 8290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:30.581734 8290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "a07753878eee4f728c49d6606998adf3"
format_stamp: "Formatted at 2026-06-20 12:25:30 on dist-test-slave-bwpp"
server_key: "379ab48f28805c99032aa7959dee7024"
server_key_iv: "68bd5aad64a7fdc0fdfa998a227d4d7f"
server_key_version: "encryptionkey@0"
I20260620 12:25:30.581816 8290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:30.591760 8290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:30.594509 8290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:30.594712 8290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:30.596030 8290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:30.596107 8290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:30.596174 8290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:30.596220 8290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:30.607324 8290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38475
I20260620 12:25:30.607343 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38475 every 8 connection(s)
I20260620 12:25:30.608290 8290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:30.611866 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.596897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47409 (local address 127.0.98.62:44045)
0620 12:25:30.597023 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.597026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.598055 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 12:25:30.599182 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.599188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.599273 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.599337 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.601163 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.602010 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.604027 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.604314 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.604409 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.607821 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.607825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.607828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.607865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.609710 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.610301 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.610304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.610305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.610384 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.610726 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.610729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.610731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.610982 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.611060 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.611634 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.611741 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:25:30.612753 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.597428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44045 (local address 127.0.98.3:47409)
0620 12:25:30.597910 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.597950 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.598891 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.599810 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.599822 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.600258 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:30.600952 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.600975 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.602168 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.602176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.603865 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.603880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.604510 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.604516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.604759 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.605502 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:30.605530 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:30.607642 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:30.609841 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.609846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.609865 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.610191 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.610465 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.610468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.610469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.610603 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.611071 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:30.611076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:30.611359 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.611538 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.611889 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260620 12:25:30.614095 8421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44045
I20260620 12:25:30.614240 8421 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:30.614876 8421 heartbeater.cc:511] Master 127.0.98.62:44045 requested a full tablet report, sending...
I20260620 12:25:30.616037 7938 ts_manager.cc:194] Registered new tserver with Master: a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475)
I20260620 12:25:30.616758 7938 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47409
I20260620 12:25:30.617625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 8290
I20260620 12:25:30.617769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:30.618091 392 external_mini_cluster.cc:1428] Setting key 1db09ea502aa76b329008dbfb7c45a0e
I20260620 12:25:30.620253 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:25:30.624512 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.616647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35975 (local address 127.0.98.62:44045)
0620 12:25:30.616778 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.616782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.616914 (+ 132us) server_negotiation.cc:407] Connection header received
0620 12:25:30.617079 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.617082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.617129 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.617213 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.617646 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.618298 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.619477 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.619773 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.619892 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.621176 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.621179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.621182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.621202 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.622941 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.623348 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.623352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.623354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.623447 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.623753 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.623756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.623757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.623921 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.623997 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.624307 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.624398 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:25:30.624843 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.616519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44045 (local address 127.0.98.3:35975)
0620 12:25:30.616829 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.616845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.616982 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.617229 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.617232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.617434 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:30.617534 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.617539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.618419 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.618422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.619346 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.619356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.619890 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.619893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.619986 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.620577 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:30.620592 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:30.621047 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:30.623048 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.623051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.623052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.623252 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.623539 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.623542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.623543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.623662 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.624023 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:30.624029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:30.624144 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.624263 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.624432 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:25:30.644891 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:30.638789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.0.98.62:44045)
0620 12:25:30.638914 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.638918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.638930 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:25:30.638969 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.638971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.639011 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.639075 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.639810 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.640317 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.641142 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.641348 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.641423 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.644706 (+ 3283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33918: BlockingRecv error: recv got EOF from 127.0.0.1:33918 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:30.654557 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.646476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.0.98.62:44045)
0620 12:25:30.646614 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.646617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.646692 (+ 75us) server_negotiation.cc:407] Connection header received
0620 12:25:30.646838 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.646841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.646882 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.646943 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.647557 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.648065 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.648904 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.649111 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.649190 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.651568 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.651571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.651573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.651596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.653005 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.653418 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.653421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.653422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.653494 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.653767 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.653769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.653770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.653963 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.654063 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.654257 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.654343 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:25:30.657701 7938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:30.661301 7938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:30.676275 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.671308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.0.98.3:38475)
0620 12:25:30.671712 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.671721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.671740 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:25:30.671799 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.671802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.671982 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.672140 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:30.672328 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.673425 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.674864 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.675527 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.675664 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.675852 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.675989 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.676098 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":120,"threads_started":1}
I20260620 12:25:30.676265 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.670988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.0.1:45280)
0620 12:25:30.671220 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.671234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.671317 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.672114 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.672117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.672136 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:30.672228 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.672234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.673589 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.673592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.674728 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.674741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.675661 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.675665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.675774 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.675811 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.675951 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.676101 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":30,"thread_start_us":83,"threads_started":1}
I20260620 12:25:30.676270 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.670643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36885 (local address 127.0.0.1:47174)
0620 12:25:30.670905 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.670920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.671014 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.671915 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.671918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.671936 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:30.672062 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.672073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.673516 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.673520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.674832 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.674844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.675821 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.675825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.675911 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.675938 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.676003 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.676107 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":28,"thread_start_us":98,"threads_started":1}
I20260620 12:25:30.676903 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.669893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33063 (local address 127.0.0.1:42912)
0620 12:25:30.670370 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.670418 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.670559 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.671640 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.671647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.671672 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:30.671852 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.671874 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.673139 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.673145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.674620 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.674631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.676001 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.676004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.676091 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.676109 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.676172 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.676230 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":102,"threads_started":1}
I20260620 12:25:30.677060 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.670856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.0.98.1:36885)
0620 12:25:30.671446 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.671463 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.671487 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:25:30.671565 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.671570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.671768 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.671942 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:30.672195 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.673389 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.674963 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.675691 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.676499 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.676745 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.676803 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.676899 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":61,"threads_started":1}
I20260620 12:25:30.677240 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.670057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.0.98.2:33063)
0620 12:25:30.670540 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.670551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.670598 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:25:30.670755 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.670761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.671031 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.671203 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:30.672118 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.673014 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.674766 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.675648 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.675773 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.675942 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.676336 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.676724 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":124,"threads_started":1}
I20260620 12:25:30.679793 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 7a05f964144c4b8691ce33373fc0be19 (DEFAULT_TABLE table=test-table [id=1140e20f258147579808c3da955d8044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:30.679795 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 7a05f964144c4b8691ce33373fc0be19 (DEFAULT_TABLE table=test-table [id=1140e20f258147579808c3da955d8044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:30.679792 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 7a05f964144c4b8691ce33373fc0be19 (DEFAULT_TABLE table=test-table [id=1140e20f258147579808c3da955d8044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:30.680828 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a05f964144c4b8691ce33373fc0be19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.680828 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a05f964144c4b8691ce33373fc0be19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.680851 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a05f964144c4b8691ce33373fc0be19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.688216 8449 tablet_bootstrap.cc:492] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Bootstrap starting.
I20260620 12:25:30.689179 8450 tablet_bootstrap.cc:492] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Bootstrap starting.
I20260620 12:25:30.690582 8449 tablet_bootstrap.cc:654] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.690965 8450 tablet_bootstrap.cc:654] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.691100 8451 tablet_bootstrap.cc:492] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Bootstrap starting.
I20260620 12:25:30.691656 8449 log.cc:824] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:30.691903 8450 log.cc:824] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:30.693475 8451 tablet_bootstrap.cc:654] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.693636 8449 tablet_bootstrap.cc:492] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: No bootstrap required, opened a new log
I20260620 12:25:30.693842 8449 ts_tablet_manager.cc:1397] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 12:25:30.693989 8450 tablet_bootstrap.cc:492] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: No bootstrap required, opened a new log
I20260620 12:25:30.694181 8450 ts_tablet_manager.cc:1397] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:25:30.694312 8451 log.cc:824] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:30.696584 8451 tablet_bootstrap.cc:492] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: No bootstrap required, opened a new log
I20260620 12:25:30.696774 8451 ts_tablet_manager.cc:1397] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:25:30.697256 8449 raft_consensus.cc:348] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.697515 8449 raft_consensus.cc:374] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.697572 8449 raft_consensus.cc:729] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07753878eee4f728c49d6606998adf3, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.698035 8449 consensus_queue.cc:260] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.698458 8450 raft_consensus.cc:348] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.698742 8450 raft_consensus.cc:374] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.698798 8450 raft_consensus.cc:729] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa2d938366645bfacbc42f40ba173fc, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.699108 8449 ts_tablet_manager.cc:1428] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 12:25:30.699237 8450 consensus_queue.cc:260] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.699469 8421 heartbeater.cc:503] Master 127.0.98.62:44045 was elected leader, sending a full tablet report...
I20260620 12:25:30.700243 8450 ts_tablet_manager.cc:1428] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:25:30.700599 8141 heartbeater.cc:503] Master 127.0.98.62:44045 was elected leader, sending a full tablet report...
I20260620 12:25:30.701040 8451 raft_consensus.cc:348] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.701264 8451 raft_consensus.cc:374] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.701329 8451 raft_consensus.cc:729] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcb8c883b8496c8ea38ae6db6214dd, State: Initialized, Role: FOLLOWER
W20260620 12:25:30.701557 8142 tablet.cc:2369] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:30.701849 8451 consensus_queue.cc:260] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.702486 8281 heartbeater.cc:503] Master 127.0.98.62:44045 was elected leader, sending a full tablet report...
I20260620 12:25:30.703961 8451 ts_tablet_manager.cc:1428] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:25:30.708388 8456 raft_consensus.cc:482] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:30.708521 8456 raft_consensus.cc:504] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.709424 8456 leader_election.cc:288] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063), a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475)
I20260620 12:25:30.713956 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.709785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33401 (local address 127.0.98.2:33063)
0620 12:25:30.709906 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.709910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.710136 (+ 226us) server_negotiation.cc:407] Connection header received
0620 12:25:30.710455 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.710460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.710523 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.710608 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:30.711043 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.711698 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.712893 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.713321 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.713452 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.713523 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.713732 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.713812 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:25:30.714128 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.710316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53531 (local address 127.0.98.3:38475)
0620 12:25:30.710483 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.710488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.710710 (+ 222us) server_negotiation.cc:407] Connection header received
0620 12:25:30.710898 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.710902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.710952 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.711042 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:30.711332 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.711956 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.713042 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.713589 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.713703 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.713759 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.713955 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.714017 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 12:25:30.714144 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.710199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.98.1:53531)
0620 12:25:30.710595 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.710611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.710741 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.711102 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.711105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.711123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:30.711227 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.711232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.712067 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.712073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.712892 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.712901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.713703 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.713707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.713805 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.713827 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.713942 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.714008 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":200,"threads_started":1}
I20260620 12:25:30.713987 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.709702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33063 (local address 127.0.98.1:33401)
0620 12:25:30.710022 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.710039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.710155 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.710620 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.710622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.710640 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:30.710898 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.710939 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.711815 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.711822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.712747 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.712767 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.713458 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.713462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.713571 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.713599 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.713732 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.713816 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"spinlock_wait_cycles":3456,"thread_start_us":128,"threads_started":1}
I20260620 12:25:30.714793 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a05f964144c4b8691ce33373fc0be19" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07753878eee4f728c49d6606998adf3" is_pre_election: true
I20260620 12:25:30.714812 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a05f964144c4b8691ce33373fc0be19" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" is_pre_election: true
I20260620 12:25:30.715090 8235 raft_consensus.cc:2454] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 0.
I20260620 12:25:30.715090 8375 raft_consensus.cc:2454] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 0.
I20260620 12:25:30.715538 8029 leader_election.cc:302] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bcb8c883b8496c8ea38ae6db6214dd, 9aa2d938366645bfacbc42f40ba173fc; no voters:
I20260620 12:25:30.715770 8456 raft_consensus.cc:2781] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:30.715852 8456 raft_consensus.cc:482] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:30.715906 8456 raft_consensus.cc:3037] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:30.716945 8456 raft_consensus.cc:504] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.717298 8456 leader_election.cc:288] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 election: Requested vote from peers 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063), a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475)
I20260620 12:25:30.717649 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a05f964144c4b8691ce33373fc0be19" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd"
I20260620 12:25:30.717765 8235 raft_consensus.cc:3037] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:30.718827 8235 raft_consensus.cc:2454] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 1.
I20260620 12:25:30.718815 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a05f964144c4b8691ce33373fc0be19" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07753878eee4f728c49d6606998adf3"
I20260620 12:25:30.718953 8375 raft_consensus.cc:3037] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:30.719131 8029 leader_election.cc:302] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bcb8c883b8496c8ea38ae6db6214dd, 9aa2d938366645bfacbc42f40ba173fc; no voters:
I20260620 12:25:30.719296 8456 raft_consensus.cc:2781] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:30.719465 8456 raft_consensus.cc:686] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 1 LEADER]: Becoming Leader. State: Replica: 9aa2d938366645bfacbc42f40ba173fc, State: Running, Role: LEADER
I20260620 12:25:30.719791 8456 consensus_queue.cc:237] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.720376 8375 raft_consensus.cc:2454] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 1.
I20260620 12:25:30.722992 7937 catalog_manager.cc:5495] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9aa2d938366645bfacbc42f40ba173fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } health_report { overall_health: UNKNOWN } } }
I20260620 12:25:30.738544 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.734758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.0.98.1:36885)
0620 12:25:30.734907 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.734912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.734927 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:30.735119 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.735122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.735184 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.735357 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:30.735604 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.736119 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.736998 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.737229 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.737351 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.737648 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:30.738116 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:30.738253 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.738322 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.738387 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:25:30.745081 8375 raft_consensus.cc:1261] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Refusing update from remote peer 9aa2d938366645bfacbc42f40ba173fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:30.745081 8235 raft_consensus.cc:1261] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Refusing update from remote peer 9aa2d938366645bfacbc42f40ba173fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:30.745782 8456 consensus_queue.cc:1035] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:30.746028 8460 consensus_queue.cc:1035] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:30.755992 8464 mvcc.cc:205] Tried to move back new op lower bound from 7298901322722500608 to 7298901322631151616. Current Snapshot: MvccSnapshot[applied={T|T < 7298901322722500608 or (T in {7298901322722500608})}]
I20260620 12:25:30.762588 7937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:30.762797 7937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:30.765270 7937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Sending DeleteTablet for 3 replicas of tablet 7a05f964144c4b8691ce33373fc0be19
I20260620 12:25:30.766010 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a05f964144c4b8691ce33373fc0be19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42912
I20260620 12:25:30.766072 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a05f964144c4b8691ce33373fc0be19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47174
I20260620 12:25:30.766152 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a05f964144c4b8691ce33373fc0be19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45280
I20260620 12:25:30.766525 8474 tablet_replica.cc:330] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: stopping tablet replica
I20260620 12:25:30.766772 8475 tablet_replica.cc:330] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: stopping tablet replica
I20260620 12:25:30.766857 8474 raft_consensus.cc:2229] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:30.767060 8473 tablet_replica.cc:330] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: stopping tablet replica
I20260620 12:25:30.767256 8474 raft_consensus.cc:2258] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:30.767275 7937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:30.767684 7937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:30.767944 8473 raft_consensus.cc:2229] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:30.768092 8475 raft_consensus.cc:2229] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:30.768667 8474 ts_tablet_manager.cc:1905] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:30.769044 8475 raft_consensus.cc:2258] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:30.769356 8473 raft_consensus.cc:2258] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:30.770864 8475 ts_tablet_manager.cc:1905] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:30.772763 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 3f29c5ba6c4f426b8887bfa9040852c9 (DEFAULT_TABLE table=test-table [id=c2f4cfd0ea114991b3de76a3fd355c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:30.772884 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 3f29c5ba6c4f426b8887bfa9040852c9 (DEFAULT_TABLE table=test-table [id=c2f4cfd0ea114991b3de76a3fd355c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:30.773018 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f29c5ba6c4f426b8887bfa9040852c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.773108 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f29c5ba6c4f426b8887bfa9040852c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.773514 8475 ts_tablet_manager.cc:1918] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:30.773597 8475 log.cc:1196] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/7a05f964144c4b8691ce33373fc0be19
I20260620 12:25:30.773895 8475 ts_tablet_manager.cc:1939] T 7a05f964144c4b8691ce33373fc0be19 P a07753878eee4f728c49d6606998adf3: Deleting consensus metadata
I20260620 12:25:30.775094 7925 catalog_manager.cc:4841] TS a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475): tablet 7a05f964144c4b8691ce33373fc0be19 (table test-table [id=1140e20f258147579808c3da955d8044]) successfully deleted
I20260620 12:25:30.775199 8449 tablet_bootstrap.cc:492] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Bootstrap starting.
I20260620 12:25:30.775341 8450 tablet_bootstrap.cc:492] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: Bootstrap starting.
I20260620 12:25:30.775455 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 3f29c5ba6c4f426b8887bfa9040852c9 (DEFAULT_TABLE table=test-table [id=c2f4cfd0ea114991b3de76a3fd355c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:30.775714 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f29c5ba6c4f426b8887bfa9040852c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.776175 8449 tablet_bootstrap.cc:654] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.776268 8450 tablet_bootstrap.cc:654] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.777267 8449 tablet_bootstrap.cc:492] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: No bootstrap required, opened a new log
I20260620 12:25:30.777346 8449 ts_tablet_manager.cc:1397] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:30.777814 8473 ts_tablet_manager.cc:1905] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:30.777751 8449 raft_consensus.cc:348] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.777866 8449 raft_consensus.cc:374] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.777899 8449 raft_consensus.cc:729] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07753878eee4f728c49d6606998adf3, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.778021 8449 consensus_queue.cc:260] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.778139 8451 tablet_bootstrap.cc:492] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Bootstrap starting.
I20260620 12:25:30.778299 8449 ts_tablet_manager.cc:1428] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:30.779037 8451 tablet_bootstrap.cc:654] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.780267 8473 ts_tablet_manager.cc:1918] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:30.780367 8473 log.cc:1196] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/7a05f964144c4b8691ce33373fc0be19
I20260620 12:25:30.780689 8473 ts_tablet_manager.cc:1939] T 7a05f964144c4b8691ce33373fc0be19 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting consensus metadata
I20260620 12:25:30.780975 8474 ts_tablet_manager.cc:1918] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:30.781064 8474 log.cc:1196] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/7a05f964144c4b8691ce33373fc0be19
I20260620 12:25:30.781350 8474 ts_tablet_manager.cc:1939] T 7a05f964144c4b8691ce33373fc0be19 P 9aa2d938366645bfacbc42f40ba173fc: Deleting consensus metadata
I20260620 12:25:30.781863 7925 catalog_manager.cc:4841] TS 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063): tablet 7a05f964144c4b8691ce33373fc0be19 (table test-table [id=1140e20f258147579808c3da955d8044]) successfully deleted
I20260620 12:25:30.782315 7925 catalog_manager.cc:4841] TS 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885): tablet 7a05f964144c4b8691ce33373fc0be19 (table test-table [id=1140e20f258147579808c3da955d8044]) successfully deleted
I20260620 12:25:30.782589 8450 tablet_bootstrap.cc:492] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: No bootstrap required, opened a new log
I20260620 12:25:30.782696 8450 ts_tablet_manager.cc:1397] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 12:25:30.783171 8450 raft_consensus.cc:348] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.783311 8450 raft_consensus.cc:374] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.783363 8450 raft_consensus.cc:729] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa2d938366645bfacbc42f40ba173fc, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.783499 8450 consensus_queue.cc:260] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.783792 8450 ts_tablet_manager.cc:1428] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:25:30.787403 8451 tablet_bootstrap.cc:492] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: No bootstrap required, opened a new log
I20260620 12:25:30.787492 8451 ts_tablet_manager.cc:1397] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260620 12:25:30.787914 8451 raft_consensus.cc:348] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.788023 8451 raft_consensus.cc:374] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.788054 8451 raft_consensus.cc:729] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcb8c883b8496c8ea38ae6db6214dd, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.788177 8451 consensus_queue.cc:260] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.788431 8451 ts_tablet_manager.cc:1428] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260620 12:25:30.859632 8422 tablet.cc:2369] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:30.867224 8455 raft_consensus.cc:482] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:30.867362 8455 raft_consensus.cc:504] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.868583 8455 leader_election.cc:288] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885), 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063)
I20260620 12:25:30.873194 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.868867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36885 (local address 127.0.98.3:38815)
0620 12:25:30.869011 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.869028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.869130 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.869577 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.869580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.869605 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:30.869745 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.869753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.870687 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.870692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.871840 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.871851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.872703 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.872707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.872835 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.872875 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.872949 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.873033 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260620 12:25:30.873641 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.869314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36261 (local address 127.0.98.2:33063)
0620 12:25:30.869433 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.869437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.869669 (+ 232us) server_negotiation.cc:407] Connection header received
0620 12:25:30.869763 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.869766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.869817 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.869901 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:30.870213 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.870978 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.872306 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.872764 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.872895 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.872980 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.873428 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.873518 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:25:30.873673 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.868945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38815 (local address 127.0.98.1:36885)
0620 12:25:30.869096 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.869101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.869118 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:25:30.869306 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.869313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.869369 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.869449 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:30.869881 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.870545 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.871991 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.872568 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.872685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.872751 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.873483 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.873557 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:25:30.874142 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.869165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33063 (local address 127.0.98.3:36261)
0620 12:25:30.869562 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.869577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.869669 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.869927 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.869929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.869950 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:30.870089 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.870097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.871109 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.871113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.872177 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.872184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.873052 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.873056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.873145 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.873167 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.873907 (+ 740us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.873970 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":111,"threads_started":1}
I20260620 12:25:30.874246 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f29c5ba6c4f426b8887bfa9040852c9" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa2d938366645bfacbc42f40ba173fc" is_pre_election: true
I20260620 12:25:30.874514 8095 raft_consensus.cc:2454] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 0.
I20260620 12:25:30.874758 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f29c5ba6c4f426b8887bfa9040852c9" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" is_pre_election: true
I20260620 12:25:30.874927 8235 raft_consensus.cc:2454] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 0.
I20260620 12:25:30.875006 8309 leader_election.cc:302] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa2d938366645bfacbc42f40ba173fc, a07753878eee4f728c49d6606998adf3; no voters:
I20260620 12:25:30.875260 8455 raft_consensus.cc:2781] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:30.875332 8455 raft_consensus.cc:482] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:30.875376 8455 raft_consensus.cc:3037] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:30.876717 8455 raft_consensus.cc:504] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.877090 8455 leader_election.cc:288] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885), 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063)
I20260620 12:25:30.877452 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f29c5ba6c4f426b8887bfa9040852c9" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa2d938366645bfacbc42f40ba173fc"
I20260620 12:25:30.877600 8095 raft_consensus.cc:3037] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:30.877583 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f29c5ba6c4f426b8887bfa9040852c9" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd"
I20260620 12:25:30.877691 8235 raft_consensus.cc:3037] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:30.878822 8095 raft_consensus.cc:2454] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 1.
I20260620 12:25:30.878806 8235 raft_consensus.cc:2454] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 1.
I20260620 12:25:30.879114 8309 leader_election.cc:302] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa2d938366645bfacbc42f40ba173fc, a07753878eee4f728c49d6606998adf3; no voters:
I20260620 12:25:30.879339 8455 raft_consensus.cc:2781] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:30.879554 8455 raft_consensus.cc:686] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 1 LEADER]: Becoming Leader. State: Replica: a07753878eee4f728c49d6606998adf3, State: Running, Role: LEADER
I20260620 12:25:30.879802 8455 consensus_queue.cc:237] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.882220 7936 catalog_manager.cc:5495] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a07753878eee4f728c49d6606998adf3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } health_report { overall_health: HEALTHY } } }
W20260620 12:25:30.903247 8282 tablet.cc:2369] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:25:30 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958329, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:30.929148 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.920690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.0.98.62:44045)
0620 12:25:30.920815 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:30.920820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:30.921020 (+ 200us) server_negotiation.cc:407] Connection header received
0620 12:25:30.921103 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:30.921105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:30.921145 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:30.921218 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:30.921991 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.922501 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.923391 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.923609 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.923694 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.926049 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:30.926052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:30.926054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:30.926075 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:30.927590 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.928008 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.928010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.928012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.928082 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:30.928377 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:30.928380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:30.928381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:30.928559 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:30.928651 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:30.928922 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 12:25:30.929015 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:25:30.929153 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:30.920548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44045 (local address 127.0.0.1:33938)
0620 12:25:30.920890 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:30.920906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:30.921024 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:30.921261 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:30.921264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:30.921746 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:30.921869 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.921876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.922613 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.922617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.923241 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:30.923249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:30.923724 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:30.923727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:30.923788 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:30.924321 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:30.924337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:30.925910 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:30.927685 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.927693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.927698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.927919 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.928162 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:30.928164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:30.928166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:30.928242 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:30.928657 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:30.928662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:30.928762 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 12:25:30.928853 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:25:30.928966 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260620 12:25:30.933372 7936 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:25:30.934114 7936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:30.938997 8075 tablet_service.cc:1467] Processing CreateTablet for tablet b724f1d32fbe45299402f9e961238fa6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a09006b1497f420d859593a66fd18d3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:25:30.938999 8215 tablet_service.cc:1467] Processing CreateTablet for tablet b724f1d32fbe45299402f9e961238fa6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a09006b1497f420d859593a66fd18d3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:25:30.939235 8341 tablet_service.cc:1467] Processing CreateTablet for tablet b724f1d32fbe45299402f9e961238fa6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a09006b1497f420d859593a66fd18d3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:25:30.939407 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b724f1d32fbe45299402f9e961238fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.939409 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b724f1d32fbe45299402f9e961238fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.939559 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b724f1d32fbe45299402f9e961238fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:30.941635 8450 tablet_bootstrap.cc:492] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: Bootstrap starting.
I20260620 12:25:30.942346 8450 tablet_bootstrap.cc:654] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.942416 8451 tablet_bootstrap.cc:492] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: Bootstrap starting.
I20260620 12:25:30.942965 8449 tablet_bootstrap.cc:492] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: Bootstrap starting.
I20260620 12:25:30.943428 8451 tablet_bootstrap.cc:654] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.943898 8449 tablet_bootstrap.cc:654] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:30.944762 8451 tablet_bootstrap.cc:492] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: No bootstrap required, opened a new log
I20260620 12:25:30.944856 8451 ts_tablet_manager.cc:1397] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 12:25:30.945334 8451 raft_consensus.cc:348] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.945469 8451 raft_consensus.cc:374] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.945525 8451 raft_consensus.cc:729] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcb8c883b8496c8ea38ae6db6214dd, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.945655 8451 consensus_queue.cc:260] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.945720 8449 tablet_bootstrap.cc:492] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: No bootstrap required, opened a new log
I20260620 12:25:30.945794 8449 ts_tablet_manager.cc:1397] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:25:30.945972 8451 ts_tablet_manager.cc:1428] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:30.945953 8457 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:30.946115 8457 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:30.946303 8449 raft_consensus.cc:348] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.946420 8449 raft_consensus.cc:374] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.946453 8449 raft_consensus.cc:729] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07753878eee4f728c49d6606998adf3, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.946581 8449 consensus_queue.cc:260] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.946892 8449 ts_tablet_manager.cc:1428] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:30.947083 8455 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:30.947206 8455 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:30.947909 8450 tablet_bootstrap.cc:492] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: No bootstrap required, opened a new log
I20260620 12:25:30.948005 8450 ts_tablet_manager.cc:1397] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 12:25:30.948449 8450 raft_consensus.cc:348] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.948606 8450 raft_consensus.cc:374] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:30.948640 8450 raft_consensus.cc:729] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa2d938366645bfacbc42f40ba173fc, State: Initialized, Role: FOLLOWER
I20260620 12:25:30.948781 8450 consensus_queue.cc:260] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:30.949046 8450 ts_tablet_manager.cc:1428] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:25:30.949039 8469 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:30.949210 8469 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:31.240713 8455 raft_consensus.cc:482] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:31.240845 8455 raft_consensus.cc:504] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.241195 8455 leader_election.cc:288] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885), 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063)
I20260620 12:25:31.241696 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b724f1d32fbe45299402f9e961238fa6" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa2d938366645bfacbc42f40ba173fc" is_pre_election: true
I20260620 12:25:31.241744 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b724f1d32fbe45299402f9e961238fa6" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" is_pre_election: true
I20260620 12:25:31.241880 8235 raft_consensus.cc:2454] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 0.
I20260620 12:25:31.241873 8095 raft_consensus.cc:2454] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 0.
I20260620 12:25:31.242173 8309 leader_election.cc:302] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa2d938366645bfacbc42f40ba173fc, a07753878eee4f728c49d6606998adf3; no voters:
I20260620 12:25:31.242410 8455 raft_consensus.cc:2781] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:31.242472 8455 raft_consensus.cc:482] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:31.242514 8455 raft_consensus.cc:3037] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.243538 8455 raft_consensus.cc:504] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.243848 8455 leader_election.cc:288] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885), 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063)
I20260620 12:25:31.244190 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b724f1d32fbe45299402f9e961238fa6" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa2d938366645bfacbc42f40ba173fc"
I20260620 12:25:31.244191 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b724f1d32fbe45299402f9e961238fa6" candidate_uuid: "a07753878eee4f728c49d6606998adf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd"
I20260620 12:25:31.244313 8235 raft_consensus.cc:3037] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.244357 8095 raft_consensus.cc:3037] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.245429 8235 raft_consensus.cc:2454] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 1.
I20260620 12:25:31.245429 8095 raft_consensus.cc:2454] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07753878eee4f728c49d6606998adf3 in term 1.
I20260620 12:25:31.245678 8309 leader_election.cc:302] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bcb8c883b8496c8ea38ae6db6214dd, a07753878eee4f728c49d6606998adf3; no voters:
I20260620 12:25:31.245857 8455 raft_consensus.cc:2781] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:31.245944 8455 raft_consensus.cc:686] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [term 1 LEADER]: Becoming Leader. State: Replica: a07753878eee4f728c49d6606998adf3, State: Running, Role: LEADER
I20260620 12:25:31.246091 8455 consensus_queue.cc:237] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.246748 8455 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: TxnStatusTablet state changed. Reason: New leader a07753878eee4f728c49d6606998adf3. Latest consensus state: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:31.246848 8455 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:25:31.247054 8492 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:25:31.247215 8492 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:25:31.247912 7936 catalog_manager.cc:5495] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a07753878eee4f728c49d6606998adf3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } health_report { overall_health: HEALTHY } } }
I20260620 12:25:31.312783 8480 consensus_queue.cc:1035] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.313349 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.308717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.0.1:45292)
0620 12:25:31.308878 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:31.308898 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:31.309004 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:31.309508 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:31.309512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:31.309522 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:25:31.309636 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.309641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.310521 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.310527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.311329 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.311339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.311918 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.311922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.312014 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.312055 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.313050 (+ 995us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.313065 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 12:25:31.313114 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 12:25:31.313201 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 12:25:31.313832 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.308976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.0.98.3:38475)
0620 12:25:31.309212 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.309215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.309225 (+ 10us) server_negotiation.cc:407] Connection header received
0620 12:25:31.309287 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.309289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.309375 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.309497 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:31.309746 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.310407 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.311486 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.311811 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.311905 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.312230 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.312934 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.313543 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.313581 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.313638 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260620 12:25:31.317608 8234 raft_consensus.cc:1261] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Refusing update from remote peer a07753878eee4f728c49d6606998adf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:31.318025 8094 raft_consensus.cc:1261] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Refusing update from remote peer a07753878eee4f728c49d6606998adf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:31.318512 8455 consensus_queue.cc:1035] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.318514 8480 consensus_queue.cc:1035] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.319394 8455 consensus_queue.cc:1035] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.320495 8501 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: TxnStatusTablet state changed. Reason: New leader a07753878eee4f728c49d6606998adf3. Latest consensus state: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:31.320655 8501 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:31.321382 8457 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: TxnStatusTablet state changed. Reason: New leader a07753878eee4f728c49d6606998adf3. Latest consensus state: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:31.321491 8457 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:31.322757 8480 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:31.322893 8480 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:25:31.324343 8505 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:31.324536 8505 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P 9aa2d938366645bfacbc42f40ba173fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:31.324790 8503 mvcc.cc:205] Tried to move back new op lower bound from 7298901325069565952 to 7298901324785369088. Current Snapshot: MvccSnapshot[applied={T|T < 7298901325069565952}]
I20260620 12:25:31.326280 8499 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:31.326393 8499 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P a07753878eee4f728c49d6606998adf3: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:25:31.328367 8457 tablet_replica.cc:439] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a07753878eee4f728c49d6606998adf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } }
I20260620 12:25:31.328483 8457 tablet_replica.cc:442] T b724f1d32fbe45299402f9e961238fa6 P 84bcb8c883b8496c8ea38ae6db6214dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:25:31.335244 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.331746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.98.3:38475)
0620 12:25:31.331854 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.331858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.332099 (+ 241us) server_negotiation.cc:407] Connection header received
0620 12:25:31.332229 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.332231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.332270 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.332325 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:31.332584 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.333063 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.334247 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.334455 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.334528 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.334838 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.334900 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.334955 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.335067 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.335108 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 12:25:31.345468 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.342134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51969 (local address 127.0.98.3:38475)
0620 12:25:31.342251 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.342256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.342399 (+ 143us) server_negotiation.cc:407] Connection header received
0620 12:25:31.342480 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.342483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.342527 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.342602 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:31.342845 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.343547 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.344352 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.344661 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.344766 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.344996 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.345069 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.345144 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.345287 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.345335 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:25:31.345498 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.342039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.98.3:51969)
0620 12:25:31.342321 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:31.342333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:31.342415 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:31.342616 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:31.342618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:31.342624 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:25:31.342747 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.342751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.343655 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.343657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.344235 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.344243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.344766 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.344769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.344825 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.344861 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.345187 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.345194 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:25:31.345253 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 12:25:31.345308 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260620 12:25:31.389995 7936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:31.390208 7936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:31.392134 7936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Sending DeleteTablet for 3 replicas of tablet 3f29c5ba6c4f426b8887bfa9040852c9
I20260620 12:25:31.392632 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f29c5ba6c4f426b8887bfa9040852c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47174
I20260620 12:25:31.392845 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f29c5ba6c4f426b8887bfa9040852c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42912
I20260620 12:25:31.393049 8517 tablet_replica.cc:330] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: stopping tablet replica
I20260620 12:25:31.393191 8517 raft_consensus.cc:2229] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:31.393321 8517 raft_consensus.cc:2258] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.393304 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f29c5ba6c4f426b8887bfa9040852c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45280
I20260620 12:25:31.393577 8519 tablet_replica.cc:330] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: stopping tablet replica
I20260620 12:25:31.393702 8519 raft_consensus.cc:2229] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:31.394009 8519 raft_consensus.cc:2258] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.394032 8517 ts_tablet_manager.cc:1905] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.396893 8517 ts_tablet_manager.cc:1918] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:31.396974 8517 log.cc:1196] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/3f29c5ba6c4f426b8887bfa9040852c9
I20260620 12:25:31.397202 8519 ts_tablet_manager.cc:1905] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.397228 8517 ts_tablet_manager.cc:1939] T 3f29c5ba6c4f426b8887bfa9040852c9 P 9aa2d938366645bfacbc42f40ba173fc: Deleting consensus metadata
I20260620 12:25:31.397466 8518 tablet_replica.cc:330] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: stopping tablet replica
I20260620 12:25:31.397581 8518 raft_consensus.cc:2229] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:31.397691 8518 raft_consensus.cc:2258] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.398115 7925 catalog_manager.cc:4841] TS 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885): tablet 3f29c5ba6c4f426b8887bfa9040852c9 (table test-table [id=c2f4cfd0ea114991b3de76a3fd355c61]) successfully deleted
I20260620 12:25:31.398265 8518 ts_tablet_manager.cc:1905] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.399770 8519 ts_tablet_manager.cc:1918] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:31.399861 8519 log.cc:1196] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/3f29c5ba6c4f426b8887bfa9040852c9
I20260620 12:25:31.400112 8519 ts_tablet_manager.cc:1939] T 3f29c5ba6c4f426b8887bfa9040852c9 P a07753878eee4f728c49d6606998adf3: Deleting consensus metadata
I20260620 12:25:31.400740 7925 catalog_manager.cc:4841] TS a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475): tablet 3f29c5ba6c4f426b8887bfa9040852c9 (table test-table [id=c2f4cfd0ea114991b3de76a3fd355c61]) successfully deleted
I20260620 12:25:31.400938 8518 ts_tablet_manager.cc:1918] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:31.401047 8518 log.cc:1196] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/3f29c5ba6c4f426b8887bfa9040852c9
I20260620 12:25:31.401374 8518 ts_tablet_manager.cc:1939] T 3f29c5ba6c4f426b8887bfa9040852c9 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting consensus metadata
I20260620 12:25:31.401980 7925 catalog_manager.cc:4841] TS 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063): tablet 3f29c5ba6c4f426b8887bfa9040852c9 (table test-table [id=c2f4cfd0ea114991b3de76a3fd355c61]) successfully deleted
Jun 20 12:25:31 dist-test-slave-bwpp krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:25:31 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:31 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:25:31.428717 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:31.421413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.98.62:44045)
0620 12:25:31.421527 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.421532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.421805 (+ 273us) server_negotiation.cc:407] Connection header received
0620 12:25:31.421938 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.421942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.421998 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.422092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:31.422952 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.423753 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.424969 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.425312 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.425418 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.428592 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33952: BlockingRecv error: recv got EOF from 127.0.0.1:33952 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 12:25:31 dist-test-slave-bwpp krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:25:31.438589 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.430183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.0.98.62:44045)
0620 12:25:31.430288 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.430292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.430533 (+ 241us) server_negotiation.cc:407] Connection header received
0620 12:25:31.430620 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.430623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.430680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.430760 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:31.431264 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.432026 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.432755 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.433090 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.433187 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.435512 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:31.435516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:31.435519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:31.435544 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:31.437098 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:31.437497 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:31.437500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:31.437502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:31.437586 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:31.437822 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:31.437825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:31.437827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:31.438030 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:31.438151 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.438331 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.438435 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:25:31.440994 7936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:31.441453 7936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:31.446139 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf824964f994d05918d229bf59e3bf6 (DEFAULT_TABLE table=test-table [id=4578873fc88f46758b7442bcdb78b8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:31.446213 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf824964f994d05918d229bf59e3bf6 (DEFAULT_TABLE table=test-table [id=4578873fc88f46758b7442bcdb78b8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:31.446467 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf824964f994d05918d229bf59e3bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:31.446516 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf824964f994d05918d229bf59e3bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:31.448688 8450 tablet_bootstrap.cc:492] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: Bootstrap starting.
I20260620 12:25:31.449028 8342 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf824964f994d05918d229bf59e3bf6 (DEFAULT_TABLE table=test-table [id=4578873fc88f46758b7442bcdb78b8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:31.449301 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf824964f994d05918d229bf59e3bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:31.449405 8450 tablet_bootstrap.cc:654] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:31.450352 8450 tablet_bootstrap.cc:492] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: No bootstrap required, opened a new log
I20260620 12:25:31.450469 8450 ts_tablet_manager.cc:1397] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:31.450862 8451 tablet_bootstrap.cc:492] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: Bootstrap starting.
I20260620 12:25:31.450992 8450 raft_consensus.cc:348] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.451104 8450 raft_consensus.cc:374] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:31.451138 8450 raft_consensus.cc:729] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa2d938366645bfacbc42f40ba173fc, State: Initialized, Role: FOLLOWER
I20260620 12:25:31.451256 8450 consensus_queue.cc:260] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.451525 8450 ts_tablet_manager.cc:1428] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:31.451768 8451 tablet_bootstrap.cc:654] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:31.451954 8533 tablet_bootstrap.cc:492] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: Bootstrap starting.
I20260620 12:25:31.453274 8533 tablet_bootstrap.cc:654] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:31.454403 8533 tablet_bootstrap.cc:492] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: No bootstrap required, opened a new log
I20260620 12:25:31.454469 8451 tablet_bootstrap.cc:492] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: No bootstrap required, opened a new log
I20260620 12:25:31.454496 8533 ts_tablet_manager.cc:1397] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 12:25:31.454545 8451 ts_tablet_manager.cc:1397] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 12:25:31.454931 8533 raft_consensus.cc:348] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.454982 8451 raft_consensus.cc:348] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.455053 8533 raft_consensus.cc:374] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:31.455084 8451 raft_consensus.cc:374] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:31.455088 8533 raft_consensus.cc:729] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07753878eee4f728c49d6606998adf3, State: Initialized, Role: FOLLOWER
I20260620 12:25:31.455127 8451 raft_consensus.cc:729] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcb8c883b8496c8ea38ae6db6214dd, State: Initialized, Role: FOLLOWER
I20260620 12:25:31.455217 8533 consensus_queue.cc:260] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.455242 8451 consensus_queue.cc:260] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.455821 8451 ts_tablet_manager.cc:1428] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:31.455912 8533 ts_tablet_manager.cc:1428] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:31.723256 8457 raft_consensus.cc:482] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:31.723412 8457 raft_consensus.cc:504] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.724650 8457 leader_election.cc:288] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885), a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475)
I20260620 12:25:31.728845 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.725384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58775 (local address 127.0.98.3:38475)
0620 12:25:31.725507 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.725511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.725640 (+ 129us) server_negotiation.cc:407] Connection header received
0620 12:25:31.725816 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.725821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.725871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.725949 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:31.726220 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.726770 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.727848 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.728248 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.728386 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.728453 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.728645 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.728712 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:31.728868 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.725258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.98.2:58775)
0620 12:25:31.725534 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:31.725551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:31.725684 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:31.725972 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:31.725974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:31.725989 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:31.726086 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.726113 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.726885 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.726890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.727720 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.727728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.728404 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.728408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.728502 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.728525 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:25:31.728629 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 12:25:31.728695 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260620 12:25:31.729125 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.724885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36885 (local address 127.0.98.2:39161)
0620 12:25:31.725190 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:31.725224 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:31.725363 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:31.725853 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:31.725856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:31.725875 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:31.726018 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.726037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.726904 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.726908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.727735 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.727744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.728486 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.728490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.728612 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.728629 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:25:31.728716 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:25:31.728801 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":22,"thread_start_us":75,"threads_started":1}
I20260620 12:25:31.729125 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.724976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39161 (local address 127.0.98.1:36885)
0620 12:25:31.725281 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.725296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.725315 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:25:31.725513 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.725518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.725577 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.725688 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:31.726156 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.726765 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.727883 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.728375 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.728485 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.728552 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.728730 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.728821 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260620 12:25:31.729271 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf824964f994d05918d229bf59e3bf6" candidate_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07753878eee4f728c49d6606998adf3" is_pre_election: true
I20260620 12:25:31.729420 8375 raft_consensus.cc:2454] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84bcb8c883b8496c8ea38ae6db6214dd in term 0.
I20260620 12:25:31.729714 8094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf824964f994d05918d229bf59e3bf6" candidate_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa2d938366645bfacbc42f40ba173fc" is_pre_election: true
I20260620 12:25:31.729799 8169 leader_election.cc:302] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bcb8c883b8496c8ea38ae6db6214dd, a07753878eee4f728c49d6606998adf3; no voters:
I20260620 12:25:31.729874 8094 raft_consensus.cc:2454] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84bcb8c883b8496c8ea38ae6db6214dd in term 0.
I20260620 12:25:31.730080 8457 raft_consensus.cc:2781] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:31.730144 8457 raft_consensus.cc:482] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:31.730196 8457 raft_consensus.cc:3037] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.731225 8457 raft_consensus.cc:504] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.731532 8457 leader_election.cc:288] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [CANDIDATE]: Term 1 election: Requested vote from peers 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885), a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475)
I20260620 12:25:31.731925 8094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf824964f994d05918d229bf59e3bf6" candidate_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa2d938366645bfacbc42f40ba173fc"
I20260620 12:25:31.731995 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf824964f994d05918d229bf59e3bf6" candidate_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07753878eee4f728c49d6606998adf3"
I20260620 12:25:31.732039 8094 raft_consensus.cc:3037] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.732090 8375 raft_consensus.cc:3037] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.733186 8094 raft_consensus.cc:2454] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84bcb8c883b8496c8ea38ae6db6214dd in term 1.
I20260620 12:25:31.733186 8375 raft_consensus.cc:2454] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84bcb8c883b8496c8ea38ae6db6214dd in term 1.
I20260620 12:25:31.733449 8169 leader_election.cc:302] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bcb8c883b8496c8ea38ae6db6214dd, a07753878eee4f728c49d6606998adf3; no voters:
I20260620 12:25:31.733594 8457 raft_consensus.cc:2781] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:31.733808 8457 raft_consensus.cc:686] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 LEADER]: Becoming Leader. State: Replica: 84bcb8c883b8496c8ea38ae6db6214dd, State: Running, Role: LEADER
I20260620 12:25:31.734068 8457 consensus_queue.cc:237] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.736344 7936 catalog_manager.cc:5495] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } health_report { overall_health: UNKNOWN } } }
I20260620 12:25:31.751103 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.747306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.0.98.2:33063)
0620 12:25:31.747603 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.747617 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.747628 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:25:31.747691 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.747693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.747741 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.747885 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:31.748145 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.748653 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.749504 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.749745 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.749876 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.750176 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.750727 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.750861 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.750899 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.750947 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260620 12:25:31.753765 8375 raft_consensus.cc:1261] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Refusing update from remote peer 84bcb8c883b8496c8ea38ae6db6214dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:31.753798 8094 raft_consensus.cc:1261] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Refusing update from remote peer 84bcb8c883b8496c8ea38ae6db6214dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:31.754315 8457 consensus_queue.cc:1035] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.754531 8539 consensus_queue.cc:1035] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.765062 7936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:31.765213 7936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:31.767094 7936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Sending DeleteTablet for 3 replicas of tablet 4bf824964f994d05918d229bf59e3bf6
I20260620 12:25:31.767776 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf824964f994d05918d229bf59e3bf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45280
I20260620 12:25:31.767810 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf824964f994d05918d229bf59e3bf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47174
I20260620 12:25:31.767977 8517 tablet_replica.cc:330] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: stopping tablet replica
I20260620 12:25:31.767978 8519 tablet_replica.cc:330] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: stopping tablet replica
I20260620 12:25:31.768088 8517 raft_consensus.cc:2229] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:31.768088 8519 raft_consensus.cc:2229] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:31.768201 8517 raft_consensus.cc:2258] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.768208 8519 raft_consensus.cc:2258] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.768841 8517 ts_tablet_manager.cc:1905] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.768872 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf824964f994d05918d229bf59e3bf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42912
I20260620 12:25:31.769033 8518 tablet_replica.cc:330] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: stopping tablet replica
I20260620 12:25:31.769129 8518 raft_consensus.cc:2229] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:31.769425 8518 raft_consensus.cc:2258] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.769552 7936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:31.769924 7936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:31.770114 8518 ts_tablet_manager.cc:1905] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.771365 8519 ts_tablet_manager.cc:1905] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.771513 8517 ts_tablet_manager.cc:1918] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:31.771582 8517 log.cc:1196] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/4bf824964f994d05918d229bf59e3bf6
I20260620 12:25:31.771832 8517 ts_tablet_manager.cc:1939] T 4bf824964f994d05918d229bf59e3bf6 P 9aa2d938366645bfacbc42f40ba173fc: Deleting consensus metadata
I20260620 12:25:31.772553 8518 ts_tablet_manager.cc:1918] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:31.772620 8518 log.cc:1196] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/4bf824964f994d05918d229bf59e3bf6
I20260620 12:25:31.772886 8518 ts_tablet_manager.cc:1939] T 4bf824964f994d05918d229bf59e3bf6 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting consensus metadata
I20260620 12:25:31.773558 7925 catalog_manager.cc:4841] TS 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885): tablet 4bf824964f994d05918d229bf59e3bf6 (table test-table [id=4578873fc88f46758b7442bcdb78b8a3]) successfully deleted
I20260620 12:25:31.773685 8519 ts_tablet_manager.cc:1918] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:31.773699 7925 catalog_manager.cc:4841] TS 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063): tablet 4bf824964f994d05918d229bf59e3bf6 (table test-table [id=4578873fc88f46758b7442bcdb78b8a3]) successfully deleted
I20260620 12:25:31.773751 8519 log.cc:1196] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/4bf824964f994d05918d229bf59e3bf6
I20260620 12:25:31.773960 8519 ts_tablet_manager.cc:1939] T 4bf824964f994d05918d229bf59e3bf6 P a07753878eee4f728c49d6606998adf3: Deleting consensus metadata
I20260620 12:25:31.774616 7925 catalog_manager.cc:4841] TS a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475): tablet 4bf824964f994d05918d229bf59e3bf6 (table test-table [id=4578873fc88f46758b7442bcdb78b8a3]) successfully deleted
I20260620 12:25:31.778323 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 05ea67ddb6734e9692fd81e898dfd846 (DEFAULT_TABLE table=test-table [id=2a4535bbf096454997379b9059b565ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:31.778564 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ea67ddb6734e9692fd81e898dfd846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:31.778879 8342 tablet_service.cc:1467] Processing CreateTablet for tablet 05ea67ddb6734e9692fd81e898dfd846 (DEFAULT_TABLE table=test-table [id=2a4535bbf096454997379b9059b565ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:31.779117 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ea67ddb6734e9692fd81e898dfd846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:31.780789 8450 tablet_bootstrap.cc:492] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc: Bootstrap starting.
I20260620 12:25:31.780803 8533 tablet_bootstrap.cc:492] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: Bootstrap starting.
I20260620 12:25:31.781025 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 05ea67ddb6734e9692fd81e898dfd846 (DEFAULT_TABLE table=test-table [id=2a4535bbf096454997379b9059b565ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:31.781234 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ea67ddb6734e9692fd81e898dfd846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:31.781600 8533 tablet_bootstrap.cc:654] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:31.781611 8450 tablet_bootstrap.cc:654] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:31.782632 8533 tablet_bootstrap.cc:492] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: No bootstrap required, opened a new log
I20260620 12:25:31.782740 8533 ts_tablet_manager.cc:1397] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:31.782775 8450 tablet_bootstrap.cc:492] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc: No bootstrap required, opened a new log
I20260620 12:25:31.782843 8450 ts_tablet_manager.cc:1397] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:31.783109 8533 raft_consensus.cc:348] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.783209 8533 raft_consensus.cc:374] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:31.783175 8450 raft_consensus.cc:348] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.783241 8533 raft_consensus.cc:729] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07753878eee4f728c49d6606998adf3, State: Initialized, Role: FOLLOWER
I20260620 12:25:31.783272 8450 raft_consensus.cc:374] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:31.783301 8450 raft_consensus.cc:729] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa2d938366645bfacbc42f40ba173fc, State: Initialized, Role: FOLLOWER
I20260620 12:25:31.783337 8533 consensus_queue.cc:260] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.783416 8450 consensus_queue.cc:260] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.783601 8533 ts_tablet_manager.cc:1428] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:25:31.783641 8450 ts_tablet_manager.cc:1428] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:31.783885 8451 tablet_bootstrap.cc:492] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: Bootstrap starting.
I20260620 12:25:31.784705 8451 tablet_bootstrap.cc:654] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:31.785756 8451 tablet_bootstrap.cc:492] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: No bootstrap required, opened a new log
I20260620 12:25:31.785821 8451 ts_tablet_manager.cc:1397] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:31.786244 8451 raft_consensus.cc:348] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.786360 8451 raft_consensus.cc:374] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:31.786391 8451 raft_consensus.cc:729] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcb8c883b8496c8ea38ae6db6214dd, State: Initialized, Role: FOLLOWER
I20260620 12:25:31.786500 8451 consensus_queue.cc:260] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.786783 8451 ts_tablet_manager.cc:1428] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:25:31.869517 8505 raft_consensus.cc:482] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:31.869673 8505 raft_consensus.cc:504] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.870179 8505 leader_election.cc:288] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063), a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475)
I20260620 12:25:31.870590 8234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ea67ddb6734e9692fd81e898dfd846" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" is_pre_election: true
I20260620 12:25:31.870709 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ea67ddb6734e9692fd81e898dfd846" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07753878eee4f728c49d6606998adf3" is_pre_election: true
I20260620 12:25:31.870790 8234 raft_consensus.cc:2454] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 0.
I20260620 12:25:31.870864 8375 raft_consensus.cc:2454] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 0.
I20260620 12:25:31.871119 8029 leader_election.cc:302] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa2d938366645bfacbc42f40ba173fc, a07753878eee4f728c49d6606998adf3; no voters:
I20260620 12:25:31.871387 8505 raft_consensus.cc:2781] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:31.871454 8505 raft_consensus.cc:482] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:31.871503 8505 raft_consensus.cc:3037] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.872503 8505 raft_consensus.cc:504] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.872864 8505 leader_election.cc:288] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 election: Requested vote from peers 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063), a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475)
I20260620 12:25:31.873255 8234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ea67ddb6734e9692fd81e898dfd846" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcb8c883b8496c8ea38ae6db6214dd"
I20260620 12:25:31.873245 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ea67ddb6734e9692fd81e898dfd846" candidate_uuid: "9aa2d938366645bfacbc42f40ba173fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07753878eee4f728c49d6606998adf3"
I20260620 12:25:31.873373 8234 raft_consensus.cc:3037] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.873373 8375 raft_consensus.cc:3037] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:31.874549 8234 raft_consensus.cc:2454] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 1.
I20260620 12:25:31.874549 8375 raft_consensus.cc:2454] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa2d938366645bfacbc42f40ba173fc in term 1.
I20260620 12:25:31.874854 8029 leader_election.cc:302] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bcb8c883b8496c8ea38ae6db6214dd, 9aa2d938366645bfacbc42f40ba173fc; no voters:
I20260620 12:25:31.875111 8505 raft_consensus.cc:2781] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:31.875380 8505 raft_consensus.cc:686] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 1 LEADER]: Becoming Leader. State: Replica: 9aa2d938366645bfacbc42f40ba173fc, State: Running, Role: LEADER
I20260620 12:25:31.875550 8505 consensus_queue.cc:237] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260620 12:25:31.877394 7938 catalog_manager.cc:5495] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9aa2d938366645bfacbc42f40ba173fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa2d938366645bfacbc42f40ba173fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } health_report { overall_health: UNKNOWN } } }
I20260620 12:25:31.897641 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.894154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.0.98.1:36885)
0620 12:25:31.894516 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.894520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.894530 (+ 10us) server_negotiation.cc:407] Connection header received
0620 12:25:31.894565 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.894568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.894608 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.894751 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:31.895010 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.895571 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.896495 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.896740 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.896869 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.897134 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.897216 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.897305 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.897404 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.897459 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260620 12:25:31.908684 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.904656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38743 (local address 127.0.98.3:38475)
0620 12:25:31.904780 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.904785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.905145 (+ 360us) server_negotiation.cc:407] Connection header received
0620 12:25:31.905322 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.905326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.905375 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.905505 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:31.905816 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.906454 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.907545 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.907818 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.907933 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.908227 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.908311 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.908399 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.908493 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.908542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 12:25:31.908703 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.904572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.98.1:38743)
0620 12:25:31.905041 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:31.905059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:31.905196 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:31.905518 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:31.905521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:31.905530 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:25:31.905677 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.905683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.906570 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.906577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.907411 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.907421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.907942 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.907946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.908038 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.908085 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.908403 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.908412 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:25:31.908482 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 12:25:31.908565 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":125,"threads_started":1}
I20260620 12:25:31.915293 8375 raft_consensus.cc:1261] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Refusing update from remote peer 9aa2d938366645bfacbc42f40ba173fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:31.915282 8234 raft_consensus.cc:1261] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Refusing update from remote peer 9aa2d938366645bfacbc42f40ba173fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:31.915720 8505 consensus_queue.cc:1035] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07753878eee4f728c49d6606998adf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.915886 8551 consensus_queue.cc:1035] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bcb8c883b8496c8ea38ae6db6214dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:31.919665 8500 mvcc.cc:205] Tried to move back new op lower bound from 7298901327521497088 to 7298901327363862528. Current Snapshot: MvccSnapshot[applied={T|T < 7298901327521497088}]
I20260620 12:25:31.937074 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.932462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36885 (local address 127.0.98.3:54965)
0620 12:25:31.933028 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:31.933044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:31.933195 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:31.933604 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:31.933608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:31.933618 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:25:31.933741 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.933759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.934806 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.934809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.935686 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:31.935698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.936281 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.936285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:31.936384 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.936430 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.936759 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.936769 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 12:25:31.936842 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 12:25:31.936930 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":65,"threads_started":1}
I20260620 12:25:31.937052 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.932726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54965 (local address 127.0.98.1:36885)
0620 12:25:31.932844 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:31.932850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:31.933115 (+ 265us) server_negotiation.cc:407] Connection header received
0620 12:25:31.933318 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:31.933322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:31.933373 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:31.933468 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:31.933897 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.934692 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.935910 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:31.936142 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:31.936266 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:31.936570 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:31.936659 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:31.936747 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:31.936852 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 12:25:31.936903 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:25:31.956511 7938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:31.956656 7938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:31.958701 7938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a668d61a111464d93fe9e6f2af23e20: Sending DeleteTablet for 3 replicas of tablet 05ea67ddb6734e9692fd81e898dfd846
I20260620 12:25:31.959225 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet 05ea67ddb6734e9692fd81e898dfd846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47174
I20260620 12:25:31.959405 8517 tablet_replica.cc:330] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc: stopping tablet replica
I20260620 12:25:31.959403 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 05ea67ddb6734e9692fd81e898dfd846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42912
I20260620 12:25:31.959537 8517 raft_consensus.cc:2229] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:31.959591 8518 tablet_replica.cc:330] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: stopping tablet replica
I20260620 12:25:31.959610 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet 05ea67ddb6734e9692fd81e898dfd846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45280
I20260620 12:25:31.959699 8518 raft_consensus.cc:2229] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:31.959729 8517 raft_consensus.cc:2258] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.959798 8519 tablet_replica.cc:330] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: stopping tablet replica
I20260620 12:25:31.959810 8518 raft_consensus.cc:2258] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.959930 8519 raft_consensus.cc:2229] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:31.960037 8519 raft_consensus.cc:2258] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:31.960597 8517 ts_tablet_manager.cc:1905] T 05ea67ddb6734e9692fd81e898dfd846 P 9aa2d938366645bfacbc42f40ba173fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.960662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 8010
I20260620 12:25:31.960716 8518 ts_tablet_manager.cc:1905] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.962996 8518 ts_tablet_manager.cc:1918] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:31.963070 8518 log.cc:1196] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/05ea67ddb6734e9692fd81e898dfd846
I20260620 12:25:31.963371 8518 ts_tablet_manager.cc:1939] T 05ea67ddb6734e9692fd81e898dfd846 P 84bcb8c883b8496c8ea38ae6db6214dd: Deleting consensus metadata
I20260620 12:25:31.963816 8519 ts_tablet_manager.cc:1905] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:31.963994 7925 catalog_manager.cc:4841] TS 84bcb8c883b8496c8ea38ae6db6214dd (127.0.98.2:33063): tablet 05ea67ddb6734e9692fd81e898dfd846 (table test-table [id=2a4535bbf096454997379b9059b565ad]) successfully deleted
I20260620 12:25:31.965677 8519 ts_tablet_manager.cc:1918] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:25:31.965726 8519 log.cc:1196] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/05ea67ddb6734e9692fd81e898dfd846
I20260620 12:25:31.965912 8519 ts_tablet_manager.cc:1939] T 05ea67ddb6734e9692fd81e898dfd846 P a07753878eee4f728c49d6606998adf3: Deleting consensus metadata
I20260620 12:25:31.966622 7925 catalog_manager.cc:4841] TS a07753878eee4f728c49d6606998adf3 (127.0.98.3:38475): tablet 05ea67ddb6734e9692fd81e898dfd846 (table test-table [id=2a4535bbf096454997379b9059b565ad]) successfully deleted
W20260620 12:25:31.971560 7925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36885 (error 108)
I20260620 12:25:31.971868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 8150
I20260620 12:25:31.973132 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:31.972792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36885 (local address 127.0.0.1:47206)
0620 12:25:31.972992 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:31.973053 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":58,"threads_started":1}
W20260620 12:25:31.973816 7925 catalog_manager.cc:4584] TS 9aa2d938366645bfacbc42f40ba173fc (127.0.98.1:36885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05ea67ddb6734e9692fd81e898dfd846: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36885: connect: Connection refused (error 111)
I20260620 12:25:31.979570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 8290
I20260620 12:25:31.987231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 7906
2026-06-20T12:25:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3453 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:32 dist-test-slave-bwpp krb5kdc[8568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:32 dist-test-slave-bwpp krb5kdc[8568](info): set up 2 sockets
Jun 20 12:25:32 dist-test-slave-bwpp krb5kdc[8568](info): commencing operation
krb5kdc: starting...
W20260620 12:25:33.035369 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:33 dist-test-slave-bwpp krb5kdc[8568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:33.155821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:33.268405 8581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:33.268692 8581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:33.268757 8581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:33.272969 8581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:33.273034 8581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:33.273053 8581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:33.273070 8581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:33.273087 8581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:33.278565 8581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:33.279687 8581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:33.280759 8581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:33.289469 8587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:33.289615 8581 server_base.cc:1034] running on GCE node
W20260620 12:25:33.289530 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:33.289466 8589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:33.290305 8581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:33.291479 8581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:33.292677 8581 hybrid_clock.cc:648] HybridClock initialized: now 1781958333292657 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T12:25:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1314 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:33 dist-test-slave-bwpp krb5kdc[8594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:33 dist-test-slave-bwpp krb5kdc[8594](info): set up 2 sockets
Jun 20 12:25:33 dist-test-slave-bwpp krb5kdc[8594](info): commencing operation
krb5kdc: starting...
W20260620 12:25:34.351470 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:34 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:34.467099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43887
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:34.574402 8607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:34.574713 8607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:34.574790 8607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:34.578697 8607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:34.578795 8607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:34.578847 8607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:34.578882 8607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:34.578920 8607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:34.583992 8607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:34.584997 8607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:34.585981 8607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:34.594575 8615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:34.594594 8612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:34.594791 8607 server_base.cc:1034] running on GCE node
W20260620 12:25:34.594596 8613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:34.595392 8607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:34.596385 8607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:34.597569 8607 hybrid_clock.cc:648] HybridClock initialized: now 1781958334597537 us; error 45 us; skew 500 ppm
Jun 20 12:25:34 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:34.600206 8607 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:34.601032 8607 webserver.cc:460] Webserver started at http://127.0.98.62:43297/ using document root <none> and password file <none>
I20260620 12:25:34.601517 8607 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:34.601575 8607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:34.601728 8607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:34.603726 8607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "9f6adf4f5e0e4329ad0d200152940daf"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "164d975db22b0b0c1f3baa886befcdc2"
server_key_iv: "7f3f77c6133ca67c9680b97833262b84"
server_key_version: "encryptionkey@0"
I20260620 12:25:34.604215 8607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f6adf4f5e0e4329ad0d200152940daf"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "164d975db22b0b0c1f3baa886befcdc2"
server_key_iv: "7f3f77c6133ca67c9680b97833262b84"
server_key_version: "encryptionkey@0"
I20260620 12:25:34.607671 8607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:25:34.609848 8622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:34.611107 8607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:34.611224 8607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "9f6adf4f5e0e4329ad0d200152940daf"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "164d975db22b0b0c1f3baa886befcdc2"
server_key_iv: "7f3f77c6133ca67c9680b97833262b84"
server_key_version: "encryptionkey@0"
I20260620 12:25:34.611325 8607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:34.630174 8607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:34.633162 8607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:34.633343 8607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:34.640326 8607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43887
I20260620 12:25:34.640352 8674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43887 every 8 connection(s)
I20260620 12:25:34.641242 8607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:34.642607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 8607
I20260620 12:25:34.642762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:34.643045 392 external_mini_cluster.cc:1428] Setting key 3c67bd7798012126351180a241c5e7e8
I20260620 12:25:34.644279 8675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:34.650688 8675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Bootstrap starting.
Jun 20 12:25:34 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:34.652938 8675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:34.653636 8675 log.cc:824] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:34.655547 8675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: No bootstrap required, opened a new log
I20260620 12:25:34.656574 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:34.644284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.0.98.62:43887)
0620 12:25:34.644700 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:34.644715 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:34.644749 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:25:34.645363 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:34.645378 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:34.645680 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:34.645944 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:34.646819 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:34.647790 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:34.648910 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:34.649349 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:34.649455 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:34.652186 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:34.652191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:34.652203 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:34.652227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:34.653960 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:34.654380 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:34.654390 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:34.654394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:34.654502 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:34.654822 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:34.654825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:34.654827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:34.655191 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:34.655392 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:34.655630 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 12:25:34.655900 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":145,"threads_started":1}
I20260620 12:25:34.658547 8675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:34.658850 8675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:34.658950 8675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6adf4f5e0e4329ad0d200152940daf, State: Initialized, Role: FOLLOWER
I20260620 12:25:34.659353 8675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:34.659468 8675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:34.659507 8675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:34.659575 8675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:34.660369 8675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:34.660642 8675 leader_election.cc:302] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f6adf4f5e0e4329ad0d200152940daf; no voters:
I20260620 12:25:34.660887 8675 leader_election.cc:288] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:34.661027 8680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:34.661260 8680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 LEADER]: Becoming Leader. State: Replica: 9f6adf4f5e0e4329ad0d200152940daf, State: Running, Role: LEADER
I20260620 12:25:34.661580 8680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:34.661988 8675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:34.663245 8682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f6adf4f5e0e4329ad0d200152940daf. Latest consensus state: current_term: 1 leader_uuid: "9f6adf4f5e0e4329ad0d200152940daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } } }
I20260620 12:25:34.663246 8681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f6adf4f5e0e4329ad0d200152940daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } } }
I20260620 12:25:34.663425 8682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:34.663460 8681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:34.663790 8690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:34.666854 8690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:34.671917 8690 catalog_manager.cc:1347] Generated new cluster ID: 174e76d13f6a4756b80d93650e676b09
I20260620 12:25:34.671991 8690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:34.690785 8690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:34.691788 8690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:34.703503 8690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Generated new TSK 0
I20260620 12:25:34.704463 8690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:25:34.739773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43887
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:34.847214 8701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:34.847455 8701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:34.847496 8701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:34.851418 8701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:34.851468 8701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:34.851523 8701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:34.856467 8701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:34.857359 8701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:34.858276 8701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:34.867640 8709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:34.867645 8707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:34.867645 8706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:34.867765 8701 server_base.cc:1034] running on GCE node
I20260620 12:25:34.868285 8701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:34.868901 8701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:34.870074 8701 hybrid_clock.cc:648] HybridClock initialized: now 1781958334870061 us; error 24 us; skew 500 ppm
Jun 20 12:25:34 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:34.876588 8701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:34.877475 8701 webserver.cc:460] Webserver started at http://127.0.98.1:44371/ using document root <none> and password file <none>
I20260620 12:25:34.878018 8701 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:34.878079 8701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:34.878310 8701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:34.880383 8701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "73f54dfde92c48b2a47393a4b07e0bc7"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "40b0c03ff6eeb1a7088cc8406e252536"
server_key_iv: "c3c8ebaedd440deca2c74322a3024374"
server_key_version: "encryptionkey@0"
I20260620 12:25:34.880945 8701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "73f54dfde92c48b2a47393a4b07e0bc7"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "40b0c03ff6eeb1a7088cc8406e252536"
server_key_iv: "c3c8ebaedd440deca2c74322a3024374"
server_key_version: "encryptionkey@0"
I20260620 12:25:34.884650 8701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:34.887048 8716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:34.888196 8701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:34.888309 8701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "73f54dfde92c48b2a47393a4b07e0bc7"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "40b0c03ff6eeb1a7088cc8406e252536"
server_key_iv: "c3c8ebaedd440deca2c74322a3024374"
server_key_version: "encryptionkey@0"
I20260620 12:25:34.888410 8701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:34.906042 8701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:34.909232 8701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:34.909433 8701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:34.910012 8701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:34.910878 8701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:34.910926 8701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:34.910991 8701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:34.911021 8701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:34.920142 8701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40495
I20260620 12:25:34.920176 8829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40495 every 8 connection(s)
I20260620 12:25:34.920991 8701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:34.925441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 8701
I20260620 12:25:34.925591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:34.925875 392 external_mini_cluster.cc:1428] Setting key 6a9aea15dcc49b8d22a6e26a440f0f1c
Jun 20 12:25:34 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:25:34.934754 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:34.922706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59097 (local address 127.0.98.62:43887)
0620 12:25:34.922837 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:34.922841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:34.923658 (+ 817us) server_negotiation.cc:407] Connection header received
0620 12:25:34.925161 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:34.925164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:34.925220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:34.925305 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:34.926748 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:34.927382 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:34.928525 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:34.928764 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:34.928875 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:34.931284 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:34.931295 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:34.931298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:34.931321 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:34.932732 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:34.933232 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:34.933235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:34.933237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:34.933318 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:34.933661 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:34.933664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:34.933666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:34.933837 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:34.933945 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:34.934419 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 12:25:34.934543 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:25:34.935503 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:34.923090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:59097)
0620 12:25:34.923517 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:34.923558 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:34.924941 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:34.925474 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:34.925481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:34.925933 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:34.926583 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:34.926599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:34.927515 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:34.927522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:34.928390 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:34.928399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:34.928912 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:34.928916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:34.929092 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:34.929686 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:34.929701 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:34.931126 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:34.932849 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:34.932855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:34.932868 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:34.933123 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:34.933411 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:34.933415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:34.933417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:34.933558 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:34.933944 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:34.933950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:34.934188 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 12:25:34.934367 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 12:25:34.934725 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260620 12:25:34.936647 8830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:34.936894 8830 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:34.937374 8830 heartbeater.cc:511] Master 127.0.98.62:43887 requested a full tablet report, sending...
I20260620 12:25:34.938822 8637 ts_manager.cc:194] Registered new tserver with Master: 73f54dfde92c48b2a47393a4b07e0bc7 (127.0.98.1:40495)
I20260620 12:25:34.939886 8637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:34.951073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43887
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:35.075942 8835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:35.076177 8835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:35.076218 8835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:35.080135 8835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:35.080183 8835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:35.080236 8835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:35.085124 8835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:35.086031 8835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:35.086964 8835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:35.095956 8840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:35.095991 8841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:35.096045 8835 server_base.cc:1034] running on GCE node
W20260620 12:25:35.095991 8843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:35.096709 8835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:35.097335 8835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:35.098528 8835 hybrid_clock.cc:648] HybridClock initialized: now 1781958335098515 us; error 25 us; skew 500 ppm
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:35.101353 8835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:35.102155 8835 webserver.cc:460] Webserver started at http://127.0.98.2:40463/ using document root <none> and password file <none>
I20260620 12:25:35.102741 8835 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:35.102805 8835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:35.103029 8835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:35.105152 8835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "8aacebcaee3f4823be6531dc383fa4f9"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f23fff3a04d87c77607aaa6f7f1a29d2"
server_key_iv: "530624849741784838d10856d31ac0f4"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.105724 8835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8aacebcaee3f4823be6531dc383fa4f9"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f23fff3a04d87c77607aaa6f7f1a29d2"
server_key_iv: "530624849741784838d10856d31ac0f4"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.109310 8835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:25:35.111737 8850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.112901 8835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:35.113018 8835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "8aacebcaee3f4823be6531dc383fa4f9"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f23fff3a04d87c77607aaa6f7f1a29d2"
server_key_iv: "530624849741784838d10856d31ac0f4"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.113116 8835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:35.127853 8835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:35.130582 8835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:35.130818 8835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:35.131407 8835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:35.132216 8835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:35.132269 8835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.132344 8835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:35.132380 8835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.141462 8835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37967
I20260620 12:25:35.141484 8963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37967 every 8 connection(s)
I20260620 12:25:35.142338 8835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:35.146586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 8835
I20260620 12:25:35.146750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:35.147059 392 external_mini_cluster.cc:1428] Setting key d815d5102ef2565d4a508045553003f8
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:25:35.156896 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:35.144140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52447 (local address 127.0.98.62:43887)
0620 12:25:35.144278 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:35.144283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:35.145137 (+ 854us) server_negotiation.cc:407] Connection header received
0620 12:25:35.146691 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:35.146694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:35.146751 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:35.146865 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:35.148096 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.148753 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.149882 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.150115 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.150223 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.152742 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:35.152749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:35.152755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:35.152789 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:35.154511 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:35.155070 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:35.155076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:35.155080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:35.155180 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:35.155556 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:35.155560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:35.155561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:35.155797 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:35.155911 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:35.156562 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 12:25:35.156703 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:35.157703 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:35.144548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:52447)
0620 12:25:35.144986 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:35.145026 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:35.146440 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:35.147031 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:35.147038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:35.147425 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:35.147960 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.147974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.148886 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.148894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.149757 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.149766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.150268 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.150271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.150432 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.151143 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:35.151159 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:35.152575 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:35.154627 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:35.154631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:35.154643 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:35.154962 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:35.155310 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:35.155313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:35.155314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:35.155451 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:35.155948 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:35.155956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:35.156290 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 12:25:35.156470 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 12:25:35.156775 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260620 12:25:35.159008 8964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:25:35.159268 8964 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:35.159875 8964 heartbeater.cc:511] Master 127.0.98.62:43887 requested a full tablet report, sending...
I20260620 12:25:35.161087 8637 ts_manager.cc:194] Registered new tserver with Master: 8aacebcaee3f4823be6531dc383fa4f9 (127.0.98.2:37967)
I20260620 12:25:35.161921 8637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:35.175047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43887
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:35.283104 8969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:35.283350 8969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:35.283392 8969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:35.287350 8969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:35.287397 8969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:35.287453 8969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:35.292691 8969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:35.293576 8969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:35.294476 8969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:35.303828 8974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:35.303856 8975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:35.303856 8977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:35.303897 8969 server_base.cc:1034] running on GCE node
I20260620 12:25:35.304430 8969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:35.305017 8969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:35.306198 8969 hybrid_clock.cc:648] HybridClock initialized: now 1781958335306185 us; error 34 us; skew 500 ppm
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:35.308775 8969 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:35.309563 8969 webserver.cc:460] Webserver started at http://127.0.98.3:46113/ using document root <none> and password file <none>
I20260620 12:25:35.310112 8969 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:35.310202 8969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:35.310424 8969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:35.312880 8969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ded724fdad44899a3f058c102e23ec8"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f27e9b63542c50b0d3457833c8d36ec6"
server_key_iv: "6c57be0a1ed707b53315a19dc1736992"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.313445 8969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ded724fdad44899a3f058c102e23ec8"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f27e9b63542c50b0d3457833c8d36ec6"
server_key_iv: "6c57be0a1ed707b53315a19dc1736992"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.317387 8969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 12:25:35.319734 8984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.320856 8969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:35.320982 8969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "3ded724fdad44899a3f058c102e23ec8"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f27e9b63542c50b0d3457833c8d36ec6"
server_key_iv: "6c57be0a1ed707b53315a19dc1736992"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.321098 8969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:35.341075 8969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:35.343871 8969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:35.344077 8969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:35.344676 8969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:35.345466 8969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:35.345537 8969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.345606 8969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:35.345655 8969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.354579 8969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44323
I20260620 12:25:35.354606 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44323 every 8 connection(s)
I20260620 12:25:35.355417 8969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:35.360829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 8969
I20260620 12:25:35.360980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:35.361253 392 external_mini_cluster.cc:1428] Setting key d854b1497e067a9af96f5219e2f944ec
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:35.369189 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:35.357175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49011 (local address 127.0.98.62:43887)
0620 12:25:35.357307 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:35.357310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:35.358116 (+ 806us) server_negotiation.cc:407] Connection header received
0620 12:25:35.359567 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:35.359570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:35.359628 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:35.359727 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:35.361093 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.361675 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.363370 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.363606 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.363686 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.365978 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:35.365981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:35.365983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:35.366005 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:35.367500 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:35.367931 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:35.367933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:35.367935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:35.368009 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:35.368289 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:35.368291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:35.368292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:35.368487 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:35.368560 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:35.368983 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 12:25:35.369074 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:25:35.370013 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:35.357558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:49011)
0620 12:25:35.357972 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:35.358021 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:35.359343 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:35.359891 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:35.359898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:35.360250 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:35.360847 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.360880 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.361807 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.361810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.363215 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.363226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.363731 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.363734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.363876 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.364416 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:35.364431 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:35.365826 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:35.367592 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:35.367596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:35.367608 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:35.367846 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:35.368087 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:35.368089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:35.368091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:35.368213 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:35.368570 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:35.368574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:35.368786 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 12:25:35.368939 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 12:25:35.369180 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260620 12:25:35.371345 9098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43887
I20260620 12:25:35.371554 9098 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:35.371979 9098 heartbeater.cc:511] Master 127.0.98.62:43887 requested a full tablet report, sending...
I20260620 12:25:35.372927 8637 ts_manager.cc:194] Registered new tserver with Master: 3ded724fdad44899a3f058c102e23ec8 (127.0.98.3:44323)
I20260620 12:25:35.373508 8637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49011
I20260620 12:25:35.375237 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 12:25:35.445202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 8701
I20260620 12:25:35.450814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 8835
I20260620 12:25:35.456485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 8969
I20260620 12:25:35.461603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 8607
I20260620 12:25:35.467835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43887
--webserver_interface=127.0.98.62
--webserver_port=43297
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:35.574162 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:35.574412 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:35.574452 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:35.578394 9107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:35.578447 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:35.578466 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:35.578483 9107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:35.578527 9107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:35.583974 9107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:35.584960 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:35.585940 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:35.594192 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:35.594223 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:35.594193 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:35.594270 9107 server_base.cc:1034] running on GCE node
I20260620 12:25:35.595089 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:35.596194 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:35.597383 9107 hybrid_clock.cc:648] HybridClock initialized: now 1781958335597371 us; error 25 us; skew 500 ppm
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:35.600075 9107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:35.600895 9107 webserver.cc:460] Webserver started at http://127.0.98.62:43297/ using document root <none> and password file <none>
I20260620 12:25:35.601438 9107 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:35.601549 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:35.605289 9107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:35.607373 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.608494 9107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:35.608639 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "9f6adf4f5e0e4329ad0d200152940daf"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "164d975db22b0b0c1f3baa886befcdc2"
server_key_iv: "7f3f77c6133ca67c9680b97833262b84"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.609154 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:35.665081 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:35.668129 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:35.668337 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:35.675734 9107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43887
I20260620 12:25:35.675763 9174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43887 every 8 connection(s)
I20260620 12:25:35.676769 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:35.680929 9175 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:25:35.682475 9175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Bootstrap starting.
I20260620 12:25:35.683964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 9107
I20260620 12:25:35.684437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40495
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43887
--webserver_port=44371
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:25:35.698591 9175 log.cc:824] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:35.704869 9175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:25:35.705201 9175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Bootstrap complete.
I20260620 12:25:35.709247 9175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:35.709645 9175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6adf4f5e0e4329ad0d200152940daf, State: Initialized, Role: FOLLOWER
I20260620 12:25:35.710112 9175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:35.710265 9175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:35.710342 9175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:35.710462 9175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:25:35.712262 9175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:35.712642 9175 leader_election.cc:302] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f6adf4f5e0e4329ad0d200152940daf; no voters:
I20260620 12:25:35.712989 9175 leader_election.cc:288] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:25:35.713073 9179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:25:35.713304 9179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [term 2 LEADER]: Becoming Leader. State: Replica: 9f6adf4f5e0e4329ad0d200152940daf, State: Running, Role: LEADER
I20260620 12:25:35.713645 9179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260620 12:25:35.714138 9175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:35.714497 9179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f6adf4f5e0e4329ad0d200152940daf. Latest consensus state: current_term: 2 leader_uuid: "9f6adf4f5e0e4329ad0d200152940daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } } }
I20260620 12:25:35.714602 9179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:35.715468 9183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:35.718493 9183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:35.719234 9183 catalog_manager.cc:1259] Loaded cluster ID: 174e76d13f6a4756b80d93650e676b09
I20260620 12:25:35.719316 9183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:35.720911 9183 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:35.721458 9183 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf: Loaded TSK: 0
I20260620 12:25:35.722321 9183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:35.718580 9180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f6adf4f5e0e4329ad0d200152940daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f6adf4f5e0e4329ad0d200152940daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } } }
I20260620 12:25:35.722859 9180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6adf4f5e0e4329ad0d200152940daf [sys.catalog]: This master's current role is: LEADER
W20260620 12:25:35.798715 9177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:35.798978 9177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:35.799052 9177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:35.803180 9177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:35.803256 9177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:35.803339 9177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:35.808355 9177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44371
--tserver_master_addrs=127.0.98.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:35.809280 9177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:35.810300 9177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:35.819712 9201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:35.819717 9204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:35.819780 9202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:35.820494 9177 server_base.cc:1034] running on GCE node
I20260620 12:25:35.820940 9177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:35.821677 9177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:35.822860 9177 hybrid_clock.cc:648] HybridClock initialized: now 1781958335822829 us; error 37 us; skew 500 ppm
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958335, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:35.826063 9177 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:25:35.826990 9177 webserver.cc:460] Webserver started at http://127.0.98.1:44371/ using document root <none> and password file <none>
I20260620 12:25:35.827530 9177 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:35.827636 9177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:35.831462 9177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:25:35.833526 9211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.834832 9177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:35.834954 9177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "73f54dfde92c48b2a47393a4b07e0bc7"
format_stamp: "Formatted at 2026-06-20 12:25:34 on dist-test-slave-bwpp"
server_key: "40b0c03ff6eeb1a7088cc8406e252536"
server_key_iv: "c3c8ebaedd440deca2c74322a3024374"
server_key_version: "encryptionkey@0"
I20260620 12:25:35.835597 9177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:35.865895 9177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:35.869130 9177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:35.869336 9177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:35.869935 9177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:35.870859 9177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:35.870929 9177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.871001 9177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:35.871044 9177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:35.879669 9177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40495
I20260620 12:25:35.879695 9324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40495 every 8 connection(s)
I20260620 12:25:35.880719 9177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:35.890633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 9177
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:35.891191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37967
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43887
--webserver_port=40463
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:35.893342 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:35.882948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:53161)
0620 12:25:35.883440 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:35.883480 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:35.884661 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:35.885667 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:35.885673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:35.886017 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:35.886454 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.886467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.887702 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.887705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.888549 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.888557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.889206 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.889209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.889344 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.889850 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:35.889865 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:35.892659 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":144,"threads_started":1}
W20260620 12:25:35.893810 9325 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43887 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43887: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260620 12:25:35.893824 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:35.882803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53161 (local address 127.0.98.62:43887)
0620 12:25:35.883264 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:35.883284 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:35.883599 (+ 315us) server_negotiation.cc:407] Connection header received
0620 12:25:35.885016 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:35.885027 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:35.885308 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:35.885587 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:35.886634 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.887590 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.888651 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.889081 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.889194 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.892930 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53161: BlockingRecv error: recv got EOF from 127.0.98.1:53161 (error 108)
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":92,"threads_started":1}
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:35.899713 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:35.894228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:37669)
0620 12:25:35.894317 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:35.894332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:35.894418 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:35.894759 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:35.894761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:35.894875 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:35.894959 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.894963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.895997 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.895999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.896517 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.896524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.897212 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.897215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.897280 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.897797 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:35.897808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:35.899627 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 12:25:35.899973 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:35.894283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37669 (local address 127.0.98.62:43887)
0620 12:25:35.894527 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:35.894533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:35.894548 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:35.894585 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:35.894590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:35.894641 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:35.894742 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:35.895060 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.895879 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.896633 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.897103 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.897229 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.899813 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37669: BlockingRecv error: recv got EOF from 127.0.98.1:37669 (error 108)
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":147}
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:35 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:35.905095 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:35.900360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:51911)
0620 12:25:35.900486 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:35.900499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:35.900581 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:35.900818 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:35.900820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:35.900964 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:35.901024 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.901028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.901707 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.901714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.902208 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:35.902215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.902742 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.902749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:35.902867 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.903309 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:35.903323 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:35.905011 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 12:25:35.905311 9325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:25:35.905506 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:35.900419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51911 (local address 127.0.98.62:43887)
0620 12:25:35.900520 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:35.900527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:35.900541 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:35.900679 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:35.900681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:35.900719 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:35.900804 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:35.901130 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.901606 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.902322 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:35.902548 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:35.902656 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:35.905154 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51911: BlockingRecv error: recv got EOF from 127.0.98.1:51911 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 12:25:36.005204 9329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:36.005443 9329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:36.005486 9329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:36.009475 9329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:36.009536 9329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:36.009593 9329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:36.014870 9329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40463
--tserver_master_addrs=127.0.98.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:36.015920 9329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:36.017009 9329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:36.026373 9337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:36.026382 9334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:36.026391 9335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:36.026450 9329 server_base.cc:1034] running on GCE node
I20260620 12:25:36.027081 9329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:36.027665 9329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:36.028823 9329 hybrid_clock.cc:648] HybridClock initialized: now 1781958336028784 us; error 55 us; skew 500 ppm
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:36.031636 9329 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:25:36.032460 9329 webserver.cc:460] Webserver started at http://127.0.98.2:40463/ using document root <none> and password file <none>
I20260620 12:25:36.033002 9329 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:36.033097 9329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:36.036947 9329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:25:36.039006 9344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:36.040199 9329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:36.040315 9329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "8aacebcaee3f4823be6531dc383fa4f9"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f23fff3a04d87c77607aaa6f7f1a29d2"
server_key_iv: "530624849741784838d10856d31ac0f4"
server_key_version: "encryptionkey@0"
I20260620 12:25:36.040846 9329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:36.061221 9329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:36.064538 9329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:36.064747 9329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:36.065368 9329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:36.066207 9329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:36.066252 9329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:36.066293 9329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:36.066357 9329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:36.074901 9329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37967
I20260620 12:25:36.074928 9457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37967 every 8 connection(s)
I20260620 12:25:36.075907 9329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:36.078783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 9329
I20260620 12:25:36.079411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44323
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43887
--webserver_port=46113
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:25:36.088560 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:36.077674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60931 (local address 127.0.98.62:43887)
0620 12:25:36.077801 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.077805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.078815 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 12:25:36.080984 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.080990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.081064 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.081167 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.082304 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.083225 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.084362 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.084780 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.084901 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.088372 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60931: BlockingRecv error: recv got EOF from 127.0.98.2:60931 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:25:36.088977 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.078059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:60931)
0620 12:25:36.078629 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:36.078709 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:36.080714 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:36.081338 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:36.081345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:36.081686 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:36.082152 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.082165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.083377 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.083381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.084240 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.084248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.084929 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.084931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.085062 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.085670 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:36.085684 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:36.088163 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
W20260620 12:25:36.089501 9458 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43887 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43887: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:36.095305 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.090049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:38905)
0620 12:25:36.090157 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:36.090171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:36.090247 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:36.090521 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:36.090523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:36.090645 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:36.090792 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.090798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.091776 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.091779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.092320 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.092327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.092861 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.092864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.092937 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.093451 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:36.093469 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:36.095220 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 12:25:36.095666 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:36.090152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38905 (local address 127.0.98.62:43887)
0620 12:25:36.090288 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.090294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.090311 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:25:36.090355 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.090359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.090407 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.090499 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.090892 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.091668 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.092421 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.092760 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.092853 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.095365 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38905: BlockingRecv error: recv got EOF from 127.0.98.2:38905 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:36.101532 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.095931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:50023)
0620 12:25:36.096050 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:36.096064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:36.096146 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:36.096354 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:36.096356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:36.096519 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:36.096605 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.096610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.097270 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.097274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.098130 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.098139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.099013 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.099017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.099104 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.099548 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:36.099561 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:36.101386 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 12:25:36.101624 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:36.095878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50023 (local address 127.0.98.62:43887)
0620 12:25:36.095991 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.095994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.096131 (+ 137us) server_negotiation.cc:407] Connection header received
0620 12:25:36.096223 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.096225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.096258 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.096321 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.096703 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.097168 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.098319 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.098782 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.098934 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.101503 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50023: BlockingRecv error: recv got EOF from 127.0.98.2:50023 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 12:25:36.101737 9458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:25:36.198448 9461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:36.198764 9461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:36.198843 9461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:36.202953 9461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:36.203004 9461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:36.203095 9461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:36.208312 9461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46113
--tserver_master_addrs=127.0.98.62:43887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:36.209242 9461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:36.210206 9461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:36.219894 9466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:36.219894 9469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:36.219884 9467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:36.220340 9461 server_base.cc:1034] running on GCE node
I20260620 12:25:36.220737 9461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:36.221323 9461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:36.222474 9461 hybrid_clock.cc:648] HybridClock initialized: now 1781958336222444 us; error 42 us; skew 500 ppm
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958336, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:36.225167 9461 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:25:36.226091 9461 webserver.cc:460] Webserver started at http://127.0.98.3:46113/ using document root <none> and password file <none>
I20260620 12:25:36.226640 9461 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:36.226796 9461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:36.230594 9461 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 12:25:36.232685 9476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:36.233955 9461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:25:36.234105 9461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "3ded724fdad44899a3f058c102e23ec8"
format_stamp: "Formatted at 2026-06-20 12:25:35 on dist-test-slave-bwpp"
server_key: "f27e9b63542c50b0d3457833c8d36ec6"
server_key_iv: "6c57be0a1ed707b53315a19dc1736992"
server_key_version: "encryptionkey@0"
I20260620 12:25:36.234647 9461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:36.255247 9461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:36.258548 9461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:36.258812 9461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:36.259413 9461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:36.260252 9461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:36.260324 9461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:36.260389 9461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:36.260439 9461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:36.269362 9461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44323
I20260620 12:25:36.269385 9589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44323 every 8 connection(s)
I20260620 12:25:36.270406 9461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:36.276443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 9461
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:25:36.282438 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:36.272246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35323 (local address 127.0.98.62:43887)
0620 12:25:36.272421 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.272425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.273316 (+ 891us) server_negotiation.cc:407] Connection header received
0620 12:25:36.274769 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.274773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.274855 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.274957 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.276100 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.276918 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.278322 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.278559 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.278697 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.282322 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35323: BlockingRecv error: recv got EOF from 127.0.98.3:35323 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 12:25:36.283248 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.272636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:35323)
0620 12:25:36.273157 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:36.273197 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:36.274521 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:36.275136 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:36.275143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:36.275492 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:36.275949 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.275962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.277095 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.277099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.278190 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.278200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.278730 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.278734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.278909 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.279544 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:36.279564 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:36.282153 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":109,"threads_started":1}
W20260620 12:25:36.283700 9590 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43887 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43887: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 12:25:36.287130 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.277450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.0.98.62:43887)
0620 12:25:36.277711 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.277714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.277724 (+ 10us) server_negotiation.cc:407] Connection header received
0620 12:25:36.277751 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.277753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.277784 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.277883 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.278615 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.279216 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.280456 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.280834 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.280946 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.282401 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:36.282408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:36.282425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:36.282455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:36.284731 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:36.285230 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:36.285237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:36.285243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:36.285378 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:36.285696 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:36.285700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:36.285702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:36.286056 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:36.286260 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:36.286608 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 12:25:36.286892 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:36.289438 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.284172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:43475)
0620 12:25:36.284372 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:36.284385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:36.284465 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:36.284719 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:36.284722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:36.284873 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:36.284954 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.284959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.285647 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.285653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.286199 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.286207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.286639 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.286642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.286759 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.287257 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:36.287276 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:36.289302 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260620 12:25:36.289521 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:36.284238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43475 (local address 127.0.98.62:43887)
0620 12:25:36.284346 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.284350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.284428 (+ 78us) server_negotiation.cc:407] Connection header received
0620 12:25:36.284578 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.284581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.284624 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.284705 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.285062 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.285540 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.286316 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.286531 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.286638 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.289401 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43475: BlockingRecv error: recv got EOF from 127.0.98.3:43475 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:36.294308 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.289974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:57449)
0620 12:25:36.290078 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:36.290095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:36.290213 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:36.290381 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:36.290383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:36.290499 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:36.290563 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.290567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.291251 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.291253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.291750 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.291757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.292146 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.292148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.292202 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.292653 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:36.292664 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:36.294224 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:25:36.294411 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:36.290054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57449 (local address 127.0.98.62:43887)
0620 12:25:36.290187 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.290190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.290202 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:25:36.290281 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.290283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.290313 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.290373 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.290691 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.291161 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.291859 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.292059 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.292129 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.294333 (+ 2204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57449: BlockingRecv error: recv got EOF from 127.0.98.3:57449 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 12:25:36.294711 9590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:36 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:36.911163 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:36.905991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:44011)
0620 12:25:36.906257 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:36.906270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:36.906355 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:36.906599 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:36.906601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:36.906790 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:36.906881 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.906886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.907669 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.907671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.908231 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:36.908239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.908674 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.908676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:36.908737 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.909193 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:36.909204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:36.911045 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260620 12:25:36.911366 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:36.906056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44011 (local address 127.0.98.62:43887)
0620 12:25:36.906317 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:36.906321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:36.906337 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:25:36.906443 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:36.906445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:36.906488 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:36.906565 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:36.907006 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.907576 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.908344 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:36.908575 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:36.908655 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:36.911188 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44011: BlockingRecv error: recv got EOF from 127.0.98.1:44011 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
Jun 20 12:25:37 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:37 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:37.108224 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:37.102344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:39381)
0620 12:25:37.102722 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:37.102738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:37.102933 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:37.103215 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:37.103218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:37.103453 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:37.103545 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:37.103550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.104309 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.104314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:37.104930 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:37.104938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.105442 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.105445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:37.105532 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:37.106080 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:37.106092 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:37.108035 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":80,"threads_started":1}
W20260620 12:25:37.108327 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:37.102471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39381 (local address 127.0.98.62:43887)
0620 12:25:37.102732 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:37.102738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:37.102855 (+ 117us) server_negotiation.cc:407] Connection header received
0620 12:25:37.103035 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:37.103038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:37.103105 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:37.103198 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:37.103664 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.104199 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.105065 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.105303 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.105438 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:37.108189 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39381: BlockingRecv error: recv got EOF from 127.0.98.2:39381 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 12:25:37 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:37 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:37.300875 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:37.295334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:58037)
0620 12:25:37.295604 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:37.295619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:37.295722 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:37.295932 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:37.295934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:37.296109 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:37.296220 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:37.296225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.296904 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.296906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:37.297890 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:37.297900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.298485 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.298489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:37.298577 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:37.299066 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:37.299077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:37.300756 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
W20260620 12:25:37.301028 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:37.295382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58037 (local address 127.0.98.62:43887)
0620 12:25:37.295520 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:37.295523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:37.295691 (+ 168us) server_negotiation.cc:407] Connection header received
0620 12:25:37.295795 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:37.295797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:37.295843 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:37.295921 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:37.296311 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.296814 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.298070 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.298371 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.298457 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:37.300881 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58037: BlockingRecv error: recv got EOF from 127.0.98.3:58037 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 12:25:37 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:37 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:37.917909 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:37.911983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:38735)
0620 12:25:37.912299 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:37.912313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:37.912412 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:37.912654 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:37.912656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:37.912831 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:37.912925 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:37.912930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.913831 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.913834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:37.914727 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:37.914739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.915204 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.915207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:37.915290 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:37.915895 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:37.915909 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:37.917759 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":114,"threads_started":1}
W20260620 12:25:37.918052 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:37.912034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38735 (local address 127.0.98.62:43887)
0620 12:25:37.912307 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:37.912311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:37.912364 (+ 53us) server_negotiation.cc:407] Connection header received
0620 12:25:37.912494 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:37.912496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:37.912540 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:37.912620 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:37.913022 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.913716 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.914856 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:37.915091 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:37.915201 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:37.917891 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38735: BlockingRecv error: recv got EOF from 127.0.98.1:38735 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
Jun 20 12:25:38 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:38 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:25:38.114936 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:38.109163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47229 (local address 127.0.98.62:43887)
0620 12:25:38.109294 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:38.109298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:38.109361 (+ 63us) server_negotiation.cc:407] Connection header received
0620 12:25:38.109485 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:38.109487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:38.109530 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:38.109602 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:38.110131 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.110833 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.111777 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.112017 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.112131 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:38.114841 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47229: BlockingRecv error: recv got EOF from 127.0.98.2:47229 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:25:38.114935 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:38.109035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:47229)
0620 12:25:38.109300 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:38.109312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:38.109406 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:38.109638 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:38.109640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:38.109885 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:38.109994 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:38.110011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.110939 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.110944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:38.111525 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:38.111533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.112134 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.112138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:38.112222 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:38.112772 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:38.112785 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:38.114714 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
Jun 20 12:25:38 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:38 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:38.307018 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:38.301656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:60171)
0620 12:25:38.301930 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:38.301945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:38.302043 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:38.302294 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:38.302296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:38.302466 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:38.302574 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:38.302578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.303329 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.303331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:38.303879 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:38.303886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.304341 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.304343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:38.304400 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:38.304912 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:38.304923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:38.306833 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
W20260620 12:25:38.307350 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:38.301854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60171 (local address 127.0.98.62:43887)
0620 12:25:38.301984 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:38.301987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:38.302000 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:25:38.302127 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:38.302129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:38.302183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:38.302260 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:38.302706 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.303234 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.304017 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.304236 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.304334 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:38.307021 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60171: BlockingRecv error: recv got EOF from 127.0.98.3:60171 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 12:25:38 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:38 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:38.929024 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:38.918729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:44187)
0620 12:25:38.919030 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:38.919048 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:38.919195 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:38.919478 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:38.919480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:38.919688 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:38.919779 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:38.919785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.920553 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.920559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:38.921125 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:38.921133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.921601 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.921604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:38.921691 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:38.926950 (+ 5259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:38.926962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:38.928831 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
W20260620 12:25:38.929097 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:38.918764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44187 (local address 127.0.98.62:43887)
0620 12:25:38.919030 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:38.919036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:38.919135 (+ 99us) server_negotiation.cc:407] Connection header received
0620 12:25:38.919292 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:38.919296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:38.919353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:38.919452 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:38.919920 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.920446 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.921243 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:38.921489 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:38.921599 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:38.928968 (+ 7369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44187: BlockingRecv error: recv got EOF from 127.0.98.1:44187 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
Jun 20 12:25:39 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:39 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:39.121551 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:39.115834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:42223)
0620 12:25:39.116104 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:39.116117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:39.116234 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:39.116554 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:39.116556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:39.116732 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:39.116828 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:39.116834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.117593 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.117596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:39.118136 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:39.118143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.118629 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.118632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:39.118752 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:39.119233 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:39.119244 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:39.121411 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
W20260620 12:25:39.121676 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:39.115859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42223 (local address 127.0.98.62:43887)
0620 12:25:39.116010 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:39.116016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:39.116238 (+ 222us) server_negotiation.cc:407] Connection header received
0620 12:25:39.116347 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:39.116349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:39.116397 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:39.116502 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:39.116950 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.117501 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.118304 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.118532 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.118612 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:39.121551 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42223: BlockingRecv error: recv got EOF from 127.0.98.2:42223 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 12:25:39 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:39 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:39.313463 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:39.307916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:34147)
0620 12:25:39.308180 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:39.308193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:39.308290 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:39.308510 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:39.308513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:39.308686 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:39.308780 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:39.308785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.309526 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.309528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:39.310125 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:39.310132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.310796 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.310799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:39.310861 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:39.311388 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:39.311399 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:39.313334 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
W20260620 12:25:39.313592 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:39.308023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34147 (local address 127.0.98.62:43887)
0620 12:25:39.308173 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:39.308178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:39.308245 (+ 67us) server_negotiation.cc:407] Connection header received
0620 12:25:39.308354 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:39.308356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:39.308403 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:39.308474 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:39.308878 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.309433 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.310251 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.310679 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.310769 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:39.313474 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34147: BlockingRecv error: recv got EOF from 127.0.98.3:34147 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 12:25:39 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:39 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:39.935276 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:39.929849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:47021)
0620 12:25:39.930124 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:39.930140 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:39.930242 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:39.930471 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:39.930473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:39.930691 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:39.930792 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:39.930798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.931521 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.931523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:39.932087 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:39.932095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.932504 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.932508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:39.932589 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:39.933190 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:39.933205 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:39.935108 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260620 12:25:39.935343 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:39.929986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47021 (local address 127.0.98.62:43887)
0620 12:25:39.930236 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:39.930240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:39.930254 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:39.930308 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:39.930311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:39.930350 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:39.930425 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:39.930904 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.931429 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.932184 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:39.932392 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:39.932467 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:39.935224 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47021: BlockingRecv error: recv got EOF from 127.0.98.1:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
Jun 20 12:25:40 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:40 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:40.128149 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:40.122249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:37843)
0620 12:25:40.122535 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:40.122551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:40.122718 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:40.122997 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:40.123000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:40.123201 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:40.123295 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:40.123301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.124056 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.124059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:40.124616 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:40.124624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.125080 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.125083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:40.125169 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:40.125726 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:40.125739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:40.127940 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
W20260620 12:25:40.128418 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:40.122407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37843 (local address 127.0.98.62:43887)
0620 12:25:40.122537 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:40.122543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:40.122638 (+ 95us) server_negotiation.cc:407] Connection header received
0620 12:25:40.122820 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:40.122825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:40.122885 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:40.122975 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:40.123407 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.123938 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.124740 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.124967 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.125075 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:40.128104 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37843: BlockingRecv error: recv got EOF from 127.0.98.2:37843 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 12:25:40 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:40 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:40.319530 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:40.314399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:37719)
0620 12:25:40.314736 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:40.314750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:40.314848 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:40.315086 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:40.315095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:40.315266 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:40.315376 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:40.315381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.316102 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.316105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:40.316696 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:40.316704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.317131 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.317134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:40.317194 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:40.317649 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:40.317659 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:40.319405 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
W20260620 12:25:40.319613 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:40.314486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37719 (local address 127.0.98.62:43887)
0620 12:25:40.314615 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:40.314641 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:40.314828 (+ 187us) server_negotiation.cc:407] Connection header received
0620 12:25:40.314912 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:40.314914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:40.314958 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:40.315052 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:40.315476 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.316011 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.316805 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.317030 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.317115 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:40.319524 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37719: BlockingRecv error: recv got EOF from 127.0.98.3:37719 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 12:25:40 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:40 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:40.941442 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:40.936120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:50403)
0620 12:25:40.936394 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:40.936407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:40.936488 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:40.936747 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:40.936749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:40.936920 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:40.937011 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:40.937016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.937823 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.937825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:40.938473 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:40.938483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.938952 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.938954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:40.939020 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:40.939624 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:40.939635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:40.941329 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
W20260620 12:25:40.941651 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:40.936163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50403 (local address 127.0.98.62:43887)
0620 12:25:40.936465 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:40.936468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:40.936482 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:40.936582 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:40.936584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:40.936637 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:40.936718 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:40.937107 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.937728 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.938608 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:40.938854 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:40.938939 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:40.941503 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50403: BlockingRecv error: recv got EOF from 127.0.98.1:50403 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
Jun 20 12:25:41 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:41 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:41.134781 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:41.129172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:57229)
0620 12:25:41.129467 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:41.129484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:41.129625 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:41.129900 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:41.129903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:41.130123 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:41.130219 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:41.130225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.131086 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.131091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:41.131684 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:41.131693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.132178 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.132181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:41.132268 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:41.132748 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:41.132758 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:41.134593 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":107,"threads_started":1}
W20260620 12:25:41.134837 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:41.129340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57229 (local address 127.0.98.62:43887)
0620 12:25:41.129472 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:41.129479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:41.129576 (+ 97us) server_negotiation.cc:407] Connection header received
0620 12:25:41.129730 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:41.129734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:41.129782 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:41.129875 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:41.130340 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.130963 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.131808 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.132056 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.132163 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:41.134735 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57229: BlockingRecv error: recv got EOF from 127.0.98.2:57229 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 12:25:41 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:41 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:41.325702 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:41.320395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:45871)
0620 12:25:41.320690 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:41.320703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:41.320793 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:41.320993 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:41.320995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:41.321180 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:41.321268 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:41.321273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.322022 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.322025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:41.322626 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:41.322635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.323098 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.323101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:41.323160 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:41.323629 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:41.323640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:41.325488 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":121,"threads_started":1}
W20260620 12:25:41.325728 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:41.320420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45871 (local address 127.0.98.62:43887)
0620 12:25:41.320578 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:41.320581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:41.320769 (+ 188us) server_negotiation.cc:407] Connection header received
0620 12:25:41.320869 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:41.320871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:41.320915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:41.320981 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:41.321360 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.321854 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.322774 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.322986 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.323074 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:41.325617 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45871: BlockingRecv error: recv got EOF from 127.0.98.3:45871 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 12:25:41 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:41 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:41.948053 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:41.942173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:50473)
0620 12:25:41.942479 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:41.942492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:41.942614 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:41.942941 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:41.942944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:41.943202 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:41.943351 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:41.943362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.944346 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.944349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:41.944910 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:41.944918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.945441 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.945445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:41.945530 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:41.946026 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:41.946037 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:41.947882 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":127,"threads_started":1}
W20260620 12:25:41.948143 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:41.942266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50473 (local address 127.0.98.62:43887)
0620 12:25:41.942509 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:41.942513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:41.942553 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:25:41.942745 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:41.942749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:41.942802 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:41.942908 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:41.943472 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.944221 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.945092 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:41.945326 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:41.945440 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:41.948017 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50473: BlockingRecv error: recv got EOF from 127.0.98.1:50473 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
Jun 20 12:25:42 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:42 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:42.141582 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:42.135680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:45549)
0620 12:25:42.136080 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:42.136097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:42.136247 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:42.136539 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:42.136542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:42.136753 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:42.136847 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:42.136853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.137611 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.137617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:42.138264 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:42.138272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.138849 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.138853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:42.138930 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:42.139531 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:42.139547 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:42.141411 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":207,"threads_started":1}
W20260620 12:25:42.141626 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:42.135926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45549 (local address 127.0.98.62:43887)
0620 12:25:42.136077 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:42.136082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:42.136189 (+ 107us) server_negotiation.cc:407] Connection header received
0620 12:25:42.136369 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:42.136372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:42.136432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:42.136520 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:42.136963 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.137499 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.138399 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.138726 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.138857 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:42.141528 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45549: BlockingRecv error: recv got EOF from 127.0.98.2:45549 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 12:25:42 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:42 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:42.331807 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:42.326558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:38839)
0620 12:25:42.326865 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:42.326879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:42.326973 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:42.327195 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:42.327197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:42.327443 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:42.327541 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:42.327547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.328266 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.328268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:42.328863 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:42.328872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.329273 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.329276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:42.329367 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:42.329838 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:42.329849 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:42.331695 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":122,"threads_started":1}
W20260620 12:25:42.331972 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:42.326613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38839 (local address 127.0.98.62:43887)
0620 12:25:42.326765 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:42.326769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:42.326943 (+ 174us) server_negotiation.cc:407] Connection header received
0620 12:25:42.327038 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:42.327040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:42.327084 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:42.327151 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:42.327644 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.328175 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.328964 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.329180 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.329254 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:42.331820 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38839: BlockingRecv error: recv got EOF from 127.0.98.3:38839 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 12:25:42 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:42 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:42.954561 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:42.948817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:58747)
0620 12:25:42.949145 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:42.949159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:42.949265 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:42.949509 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:42.949512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:42.949696 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:42.949795 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:42.949800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.950550 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.950553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:42.951147 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:42.951157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.951580 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.951583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:42.951646 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:42.952155 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:42.952167 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:42.954377 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
W20260620 12:25:42.954669 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:42.948897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58747 (local address 127.0.98.62:43887)
0620 12:25:42.949137 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:42.949141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:42.949228 (+ 87us) server_negotiation.cc:407] Connection header received
0620 12:25:42.949345 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:42.949347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:42.949389 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:42.949465 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:42.949895 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.950424 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.951255 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:42.951472 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:42.951554 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:42.954512 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58747: BlockingRecv error: recv got EOF from 127.0.98.1:58747 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
Jun 20 12:25:43 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:43 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:43.148022 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:43.142587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:33199)
0620 12:25:43.142905 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:43.142918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:43.143027 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:43.143210 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:43.143211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:43.143384 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:43.143476 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:43.143481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.144401 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.144403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:43.144956 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:43.144966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.145437 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.145440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:43.145505 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:43.145994 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:43.146006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:43.147911 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
W20260620 12:25:43.148201 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:43.142666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33199 (local address 127.0.98.62:43887)
0620 12:25:43.142782 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:43.142786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:43.142967 (+ 181us) server_negotiation.cc:407] Connection header received
0620 12:25:43.143093 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:43.143095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:43.143135 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:43.143201 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:43.143620 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.144303 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.145088 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.145335 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.145414 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:43.148041 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33199: BlockingRecv error: recv got EOF from 127.0.98.2:33199 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 12:25:43 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:43 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:43.338322 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:43.332568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:51535)
0620 12:25:43.332856 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:43.332869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:43.332984 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:43.333271 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:43.333273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:43.333457 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:43.333552 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:43.333557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.334452 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.334454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:43.335209 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:43.335218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.335662 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.335665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:43.335746 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:43.336227 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:43.336239 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:43.338155 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":66,"threads_started":1}
W20260620 12:25:43.338368 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:43.332670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51535 (local address 127.0.98.62:43887)
0620 12:25:43.332854 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:43.332858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:43.332965 (+ 107us) server_negotiation.cc:407] Connection header received
0620 12:25:43.333061 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:43.333063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:43.333109 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:43.333197 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:43.333656 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.334289 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.335329 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.335549 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.335659 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:43.338268 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51535: BlockingRecv error: recv got EOF from 127.0.98.3:51535 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 12:25:43 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:43 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:43.967247 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:43.955378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:52021)
0620 12:25:43.955681 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:43.955693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:43.955781 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:43.956045 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:43.956047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:43.956218 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:43.956312 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:43.956316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.957042 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.957044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:43.957600 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:43.957607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.958051 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.958053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:43.958112 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:43.965119 (+ 7007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:43.965132 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:43.967131 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":113,"threads_started":1}
W20260620 12:25:43.967468 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:43.955528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52021 (local address 127.0.98.62:43887)
0620 12:25:43.955741 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:43.955744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:43.955757 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:25:43.955884 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:43.955886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:43.955929 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:43.956003 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:43.956410 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.956949 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.957709 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:43.957950 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:43.958031 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:43.967297 (+ 9266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52021: BlockingRecv error: recv got EOF from 127.0.98.1:52021 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
Jun 20 12:25:44 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:44 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:44.154912 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:44.148754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:45559)
0620 12:25:44.149026 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:44.149039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:44.149129 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:44.149614 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:44.149618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:44.149851 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:44.149985 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:44.150018 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.151252 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.151258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:44.151956 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:44.151965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.152483 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.152487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:44.152567 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:44.153135 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:44.153147 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:44.154795 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
W20260620 12:25:44.154996 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:44.148809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45559 (local address 127.0.98.62:43887)
0620 12:25:44.149334 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:44.149339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:44.149365 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:25:44.149407 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:44.149410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:44.149477 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:44.149570 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:44.150133 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.151146 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.152082 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.152331 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.152480 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:44.154907 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45559: BlockingRecv error: recv got EOF from 127.0.98.2:45559 (error 108)
Metrics: {"server-negotiator.queue_time_us":413}
Jun 20 12:25:44 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:44 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:44.344779 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:44.339359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:38173)
0620 12:25:44.339605 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:44.339623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:44.339762 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:44.340050 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:44.340053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:44.340259 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:44.340352 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:44.340359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.341132 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.341137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:44.341710 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:44.341720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.342195 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.342199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:44.342288 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:44.342798 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:44.342810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:44.344668 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
W20260620 12:25:44.344946 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:44.339408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38173 (local address 127.0.98.62:43887)
0620 12:25:44.339547 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:44.339551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:44.339710 (+ 159us) server_negotiation.cc:407] Connection header received
0620 12:25:44.339869 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:44.339873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:44.339940 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:44.340021 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:44.340473 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.341010 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.341836 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.342068 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.342178 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:44.344794 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38173: BlockingRecv error: recv got EOF from 127.0.98.3:38173 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 12:25:44 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:44 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:44.973457 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:44.968063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:41535)
0620 12:25:44.968421 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:44.968435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:44.968533 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:44.968744 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:44.968746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:44.968926 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:44.969024 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:44.969029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.969748 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.969750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:44.970326 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:44.970334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.970754 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.970757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:44.970825 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:44.971295 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:44.971307 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:44.973335 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
W20260620 12:25:44.973600 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:44.968157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41535 (local address 127.0.98.62:43887)
0620 12:25:44.968425 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:44.968434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:44.968487 (+ 53us) server_negotiation.cc:407] Connection header received
0620 12:25:44.968597 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:44.968599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:44.968640 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:44.968715 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:44.969131 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.969650 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.970429 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:44.970665 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:44.970745 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:44.973466 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41535: BlockingRecv error: recv got EOF from 127.0.98.1:41535 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":54,"threads_started":1}
Jun 20 12:25:45 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:45 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:45.161566 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:45.155629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:42423)
0620 12:25:45.156049 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:45.156062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:45.156197 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:45.156516 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:45.156519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:45.156721 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:45.156813 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:45.156819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.157766 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.157770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:45.158368 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:45.158376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.158887 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.158891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:45.158979 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:45.159472 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:45.159485 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:45.161405 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":89,"threads_started":1}
W20260620 12:25:45.161664 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:45.155691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42423 (local address 127.0.98.62:43887)
0620 12:25:45.155985 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:45.155990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:45.156139 (+ 149us) server_negotiation.cc:407] Connection header received
0620 12:25:45.156298 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:45.156302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:45.156384 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:45.156479 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:45.156937 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.157632 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.158488 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.158775 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.158885 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:45.161548 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42423: BlockingRecv error: recv got EOF from 127.0.98.2:42423 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Jun 20 12:25:45 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:45 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:45.351392 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:45.345641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:43101)
0620 12:25:45.345896 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:45.345909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:45.346039 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:45.346311 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:45.346314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:45.346527 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:45.346619 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:45.346625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.347417 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.347422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:45.347987 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:45.347995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.348447 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.348451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:45.348546 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:45.349159 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:45.349179 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:45.351219 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
W20260620 12:25:45.351446 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:45.345692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43101 (local address 127.0.98.62:43887)
0620 12:25:45.345829 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:45.345834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:45.345984 (+ 150us) server_negotiation.cc:407] Connection header received
0620 12:25:45.346137 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:45.346140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:45.346194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:45.346282 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:45.346769 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.347303 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.348109 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.348334 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.348444 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:45.351339 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43101: BlockingRecv error: recv got EOF from 127.0.98.3:43101 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 12:25:45 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:45 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:45.980070 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:45.974465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:40979)
0620 12:25:45.974772 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:45.974791 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:45.974890 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:45.975165 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:45.975166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:45.975336 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:45.975426 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:45.975431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.976184 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.976186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:45.976726 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:45.976733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.977188 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.977190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:45.977247 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:45.977831 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:45.977844 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:45.979952 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
W20260620 12:25:45.980383 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:45.974597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40979 (local address 127.0.98.62:43887)
0620 12:25:45.974948 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:45.974952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:45.974965 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:25:45.975000 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:45.975002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:45.975044 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:45.975127 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:45.975536 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.976091 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.976838 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:45.977057 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:45.977187 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:45.980109 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40979: BlockingRecv error: recv got EOF from 127.0.98.1:40979 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
Jun 20 12:25:46 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:46 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:46.168030 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:46.162419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:47243)
0620 12:25:46.162710 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:46.162727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:46.162835 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:46.163118 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:46.163121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:46.163338 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:46.163430 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:46.163436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.164223 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.164228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:46.164853 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:46.164862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.165349 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.165353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:46.165442 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:46.165912 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:46.165923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:46.167916 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
W20260620 12:25:46.168330 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:46.162532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47243 (local address 127.0.98.62:43887)
0620 12:25:46.162702 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:46.162706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:46.162784 (+ 78us) server_negotiation.cc:407] Connection header received
0620 12:25:46.162945 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:46.162949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:46.162997 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:46.163092 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:46.163548 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.164108 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.164987 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.165225 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.165337 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:46.168057 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47243: BlockingRecv error: recv got EOF from 127.0.98.2:47243 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 12:25:46 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:46 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:46.357626 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:46.352293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:59327)
0620 12:25:46.352607 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:46.352620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:46.352712 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:46.352951 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:46.352953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:46.353134 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:46.353220 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:46.353225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.353899 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.353901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:46.354512 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:46.354519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.355065 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.355068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:46.355125 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:46.355714 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:46.355726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:46.357489 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
W20260620 12:25:46.357812 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:46.352495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59327 (local address 127.0.98.62:43887)
0620 12:25:46.352635 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:46.352638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:46.352670 (+ 32us) server_negotiation.cc:407] Connection header received
0620 12:25:46.352793 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:46.352795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:46.352848 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:46.352915 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:46.353310 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.353797 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.354665 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.354958 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.355053 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:46.357693 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59327: BlockingRecv error: recv got EOF from 127.0.98.3:59327 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 12:25:46 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:46 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:46.986176 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:46.980900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:40329)
0620 12:25:46.981180 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:46.981198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:46.981290 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:46.981561 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:46.981563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:46.981741 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:46.981842 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:46.981848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.982584 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.982586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:46.983170 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:46.983179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.983601 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.983604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:46.983667 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:46.984157 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:46.984170 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:46.986042 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
W20260620 12:25:46.986308 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:46.980934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40329 (local address 127.0.98.62:43887)
0620 12:25:46.981160 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:46.981163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:46.981266 (+ 103us) server_negotiation.cc:407] Connection header received
0620 12:25:46.981362 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:46.981364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:46.981417 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:46.981492 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:46.981946 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.982468 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.983273 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:46.983494 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:46.983571 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:46.986175 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40329: BlockingRecv error: recv got EOF from 127.0.98.1:40329 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
Jun 20 12:25:47 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:47 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:25:47.174364 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:47.169015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49007 (local address 127.0.98.62:43887)
0620 12:25:47.169180 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:47.169185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:47.169353 (+ 168us) server_negotiation.cc:407] Connection header received
0620 12:25:47.169524 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:47.169527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:47.169578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:47.169666 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:47.170124 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.170692 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.171518 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.171743 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.171858 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:47.174268 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49007: BlockingRecv error: recv got EOF from 127.0.98.2:49007 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 12:25:47.174374 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:47.168901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:49007)
0620 12:25:47.169228 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:47.169241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:47.169419 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:47.169690 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:47.169692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:47.169899 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:47.169990 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:47.169996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.170800 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.170804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:47.171398 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:47.171406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.171859 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.171863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:47.171950 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:47.172409 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:47.172419 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:47.174149 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
Jun 20 12:25:47 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:47 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:47.363951 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:47.358343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:34681)
0620 12:25:47.358637 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:47.358652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:47.358808 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:47.359091 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:47.359094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:47.359299 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:47.359386 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:47.359392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.360154 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.360159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:47.360728 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:47.360736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.361204 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.361207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:47.361289 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:47.361838 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:47.361852 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:47.363770 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
W20260620 12:25:47.363987 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:47.358460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34681 (local address 127.0.98.62:43887)
0620 12:25:47.358611 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:47.358616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:47.358757 (+ 141us) server_negotiation.cc:407] Connection header received
0620 12:25:47.358924 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:47.358927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:47.358982 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:47.359069 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:47.359508 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.360040 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.360846 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.361091 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.361201 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:47.363887 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34681: BlockingRecv error: recv got EOF from 127.0.98.3:34681 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 12:25:47 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:47 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:47.992395 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:47.987034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:40685)
0620 12:25:47.987297 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:47.987312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:47.987398 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:47.987644 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:47.987646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:47.987820 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:47.987911 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:47.987916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.988711 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.988714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:47.989271 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:47.989278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.989752 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.989754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:47.989813 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:47.990332 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:47.990343 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:47.992283 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260620 12:25:47.992553 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:47.987107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40685 (local address 127.0.98.62:43887)
0620 12:25:47.987378 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:47.987382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:47.987397 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:47.987481 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:47.987483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:47.987526 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:47.987609 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:47.988030 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.988609 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.989410 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:47.989644 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:47.989738 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:47.992421 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40685: BlockingRecv error: recv got EOF from 127.0.98.1:40685 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":112,"threads_started":1}
Jun 20 12:25:48 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:48 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:48.180912 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:48.175144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:50675)
0620 12:25:48.175524 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:48.175535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:48.175669 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:48.175950 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:48.175953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:48.176155 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:48.176248 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:48.176254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.177003 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.177008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:48.177563 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:48.177571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.178033 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.178037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:48.178124 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:48.178773 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:48.178785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:48.180741 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":112,"threads_started":1}
W20260620 12:25:48.180967 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:48.175226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50675 (local address 127.0.98.62:43887)
0620 12:25:48.175435 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:48.175439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:48.175608 (+ 169us) server_negotiation.cc:407] Connection header received
0620 12:25:48.175770 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:48.175773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:48.175834 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:48.175925 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:48.176371 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.176895 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.177685 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.177921 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.178028 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:48.180860 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50675: BlockingRecv error: recv got EOF from 127.0.98.2:50675 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 12:25:48 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:48 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:48.370358 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:48.364820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:46259)
0620 12:25:48.365225 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:48.365238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:48.365369 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:48.365646 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:48.365649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:48.365862 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:48.365950 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:48.365956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.366757 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.366763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:48.367322 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:48.367331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.367790 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.367794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:48.367887 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:48.368453 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:48.368463 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:48.370191 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":188,"threads_started":1}
W20260620 12:25:48.370471 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:48.364897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46259 (local address 127.0.98.62:43887)
0620 12:25:48.365137 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:48.365142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:48.365315 (+ 173us) server_negotiation.cc:407] Connection header received
0620 12:25:48.365482 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:48.365487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:48.365536 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:48.365626 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:48.366077 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.366603 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.367443 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.367683 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.367789 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:48.370314 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46259: BlockingRecv error: recv got EOF from 127.0.98.3:46259 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Jun 20 12:25:48 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:48 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:49.002974 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:48.993351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:42255)
0620 12:25:48.993665 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:48.993688 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:48.993784 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:48.994037 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:48.994039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:48.994216 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:48.994311 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:48.994353 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.995339 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.995346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:48.996098 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:48.996108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.996557 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.996559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:48.996618 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:49.000911 (+ 4293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:49.000923 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:49.002863 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
W20260620 12:25:49.003178 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:48.993515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42255 (local address 127.0.98.62:43887)
0620 12:25:48.993757 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:48.993760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:48.993776 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:25:48.993870 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:48.993872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:48.993915 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:48.993994 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:48.994485 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.995194 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.996204 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:48.996457 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:48.996538 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:49.002999 (+ 6461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42255: BlockingRecv error: recv got EOF from 127.0.98.1:42255 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
Jun 20 12:25:49 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:49 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:49.187408 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:49.181651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:39741)
0620 12:25:49.181908 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:49.181922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:49.182011 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:49.182296 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:49.182299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:49.182495 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:49.182600 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:49.182606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.183733 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.183740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:49.184373 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:49.184385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.184842 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.184855 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:49.184921 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:49.185437 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:49.185448 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:49.187295 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
W20260620 12:25:49.187593 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:49.181828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39741 (local address 127.0.98.62:43887)
0620 12:25:49.181961 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:49.181967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:49.181984 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:25:49.182093 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:49.182095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:49.182136 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:49.182206 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:49.182800 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.183594 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.184512 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.184729 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.184828 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:49.187431 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39741: BlockingRecv error: recv got EOF from 127.0.98.2:39741 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 12:25:49 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:49 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:49.377151 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:49.371251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:46799)
0620 12:25:49.371569 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:49.371583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:49.371723 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:49.372029 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:49.372033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:49.372238 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:49.372335 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:49.372341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.373107 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.373112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:49.373745 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:49.373756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.374378 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.374381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:49.374483 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:49.375111 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:49.375132 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:49.377041 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
W20260620 12:25:49.377283 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:49.371351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46799 (local address 127.0.98.62:43887)
0620 12:25:49.371537 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:49.371542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:49.371672 (+ 130us) server_negotiation.cc:407] Connection header received
0620 12:25:49.371855 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:49.371858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:49.371907 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:49.372005 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:49.372458 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.372996 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.373881 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:49.374252 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:49.374377 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:49.377168 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46799: BlockingRecv error: recv got EOF from 127.0.98.3:46799 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 12:25:50 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:50 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:50.010277 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:50.003908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:49867)
0620 12:25:50.004346 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:50.004362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:50.004473 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:50.004734 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:50.004737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:50.004932 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:50.005094 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:50.005102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.006312 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.006317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:50.007149 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:50.007158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.007601 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.007603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:50.007661 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:50.008174 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:50.008185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:50.010094 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":219,"threads_started":1}
W20260620 12:25:50.010392 9648 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:50.004008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49867 (local address 127.0.98.62:43887)
0620 12:25:50.004329 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:50.004332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:50.004463 (+ 131us) server_negotiation.cc:407] Connection header received
0620 12:25:50.004547 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:50.004549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:50.004595 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:50.004679 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:50.005234 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.006197 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.007241 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.007504 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.007586 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:50.010217 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49867: BlockingRecv error: recv got EOF from 127.0.98.1:49867 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
Jun 20 12:25:50 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:50 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:50.194103 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:50.188192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:51523)
0620 12:25:50.188501 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:50.188514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:50.188614 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:50.188844 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:50.188846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:50.189023 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:50.189119 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:50.189124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.189884 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.189887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:50.190514 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:50.190523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.191002 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.191005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:50.191082 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:50.191687 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:50.191700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:50.193974 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
W20260620 12:25:50.194211 9648 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:50.188259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51523 (local address 127.0.98.62:43887)
0620 12:25:50.188419 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:50.188423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:50.188588 (+ 165us) server_negotiation.cc:407] Connection header received
0620 12:25:50.188695 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:50.188697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:50.188741 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:50.188802 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:50.189222 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.189764 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.190625 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.190871 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.190954 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:50.194102 (+ 3148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51523: BlockingRecv error: recv got EOF from 127.0.98.2:51523 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 12:25:50 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:50 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:50.383414 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:50.377989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.3:40861)
0620 12:25:50.378293 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:50.378308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:50.378429 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:50.378719 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:50.378722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:50.378950 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:50.379046 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:50.379052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.379821 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.379826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:50.380391 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:50.380399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.380857 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.380860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:50.380945 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:50.381413 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:50.381424 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:50.383225 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
W20260620 12:25:50.383538 9648 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:50.378090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40861 (local address 127.0.98.62:43887)
0620 12:25:50.378243 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:50.378247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:50.378388 (+ 141us) server_negotiation.cc:407] Connection header received
0620 12:25:50.378528 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:50.378531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:50.378585 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:50.378695 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:50.379163 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.379696 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.380512 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:50.380739 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:50.380848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:50.383408 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40861: BlockingRecv error: recv got EOF from 127.0.98.3:40861 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 12:25:51 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:51 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:51.017097 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:51.011215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.1:57465)
0620 12:25:51.011461 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:51.011478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:51.011581 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:51.011810 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:51.011812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:51.011981 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:51.012074 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.012079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.012951 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.012954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.013597 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.013607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.014128 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.014132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.014215 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.014851 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:51.014883 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:51.016972 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
W20260620 12:25:51.017256 9652 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:51.011321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57465 (local address 127.0.98.62:43887)
0620 12:25:51.011598 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:51.011602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:51.011617 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:51.011654 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:51.011656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:51.011697 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:51.011796 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:51.012173 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.012852 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.013751 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.014011 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.014124 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.017121 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57465: BlockingRecv error: recv got EOF from 127.0.98.1:57465 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
Jun 20 12:25:51 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:25:51 dist-test-slave-bwpp krb5kdc[8594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:25:51.200774 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:51.195204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43887 (local address 127.0.98.2:39435)
0620 12:25:51.195503 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:51.195519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:51.195635 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:51.195944 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:51.195946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:51.196168 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:51.196256 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.196261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.197037 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.197043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.197609 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.197616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.198088 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.198091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.198177 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.198746 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:51.198758 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:51.200663 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43887: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
W20260620 12:25:51.200940 9652 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:51.195326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39435 (local address 127.0.98.62:43887)
0620 12:25:51.195715 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:51.195720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:51.195733 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:25:51.195788 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:51.195790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:51.195830 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:51.195920 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:51.196381 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.196923 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.197730 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.197974 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.198086 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.200815 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39435: BlockingRecv error: recv got EOF from 127.0.98.2:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":254}
I20260620 12:25:51.279325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 9177
I20260620 12:25:51.285135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 9329
I20260620 12:25:51.290486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 9461
I20260620 12:25:51.295707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 9107
2026-06-20T12:25:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17992 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T12:25:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:51Z Disabled control of system clock
I20260620 12:25:51.325861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38239
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 12:25:51.434554 9657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:51.434861 9657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:51.434897 9657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:51.438910 9657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:51.438957 9657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:51.438974 9657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:51.438992 9657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:51.439006 9657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:51.443684 9657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:51.444644 9657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:51.445886 9657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:51.454006 9663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:51.453979 9665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:51.453994 9662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:51.454159 9657 server_base.cc:1034] running on GCE node
I20260620 12:25:51.455034 9657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:51.456090 9657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:51.457247 9657 hybrid_clock.cc:648] HybridClock initialized: now 1781958351457230 us; error 32 us; skew 500 ppm
I20260620 12:25:51.459162 9657 webserver.cc:460] Webserver started at http://127.0.98.62:45371/ using document root <none> and password file <none>
I20260620 12:25:51.459733 9657 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:51.459794 9657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:51.460013 9657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:51.462020 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "08fc8b5cf7144abeb1c0873acaea2c09"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "340d97801b8a127101f8373e3b41ad7d"
server_key_iv: "bc78d310b024afe476ee266015bed469"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.462576 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "08fc8b5cf7144abeb1c0873acaea2c09"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "340d97801b8a127101f8373e3b41ad7d"
server_key_iv: "bc78d310b024afe476ee266015bed469"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.466104 9657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:25:51.468550 9671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:51.469697 9657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:51.469794 9657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "08fc8b5cf7144abeb1c0873acaea2c09"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "340d97801b8a127101f8373e3b41ad7d"
server_key_iv: "bc78d310b024afe476ee266015bed469"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.469926 9657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:51.484836 9657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:51.485476 9657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:51.485663 9657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:51.492446 9657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38239
I20260620 12:25:51.492520 9723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38239 every 8 connection(s)
I20260620 12:25:51.493377 9657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:51.496388 9724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:51.501569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 9657
I20260620 12:25:51.501681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:51.501694 9724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09: Bootstrap starting.
I20260620 12:25:51.502015 392 external_mini_cluster.cc:1428] Setting key 1e27bdaa31a0385b2bd21d14116b8757
I20260620 12:25:51.504276 9724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:51.504976 9724 log.cc:824] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:51.506795 9724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09: No bootstrap required, opened a new log
I20260620 12:25:51.510957 9724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260620 12:25:51.511196 9724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:51.511250 9724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fc8b5cf7144abeb1c0873acaea2c09, State: Initialized, Role: FOLLOWER
I20260620 12:25:51.511492 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:51.503354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.0.98.62:38239)
0620 12:25:51.503978 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:51.503991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:51.504024 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:25:51.504666 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:51.504684 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:51.505057 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:51.505414 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:51.505813 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.506814 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.507696 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.508425 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.508594 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.509355 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:51.509362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:51.509376 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:51.509955 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:51.510049 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:51.510083 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:51.510185 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:51.510311 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 12:25:51.510568 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
I20260620 12:25:51.511710 9724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260620 12:25:51.511857 9724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:51.511922 9724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:51.512012 9724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:51.512936 9724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260620 12:25:51.513301 9724 leader_election.cc:302] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08fc8b5cf7144abeb1c0873acaea2c09; no voters:
I20260620 12:25:51.513684 9724 leader_election.cc:288] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:51.513782 9729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:51.513989 9729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [term 1 LEADER]: Becoming Leader. State: Replica: 08fc8b5cf7144abeb1c0873acaea2c09, State: Running, Role: LEADER
I20260620 12:25:51.514317 9729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260620 12:25:51.514928 9724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:51.515970 9731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08fc8b5cf7144abeb1c0873acaea2c09. Latest consensus state: current_term: 1 leader_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } } }
I20260620 12:25:51.516119 9731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:51.516474 9730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fc8b5cf7144abeb1c0873acaea2c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } } }
I20260620 12:25:51.516536 9738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:51.516609 9730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:51.519394 9738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:51.523976 9738 catalog_manager.cc:1347] Generated new cluster ID: e7b5b2e1d62548979e1e70dd2bb1e44e
I20260620 12:25:51.524035 9738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:51.551301 9738 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:51.552218 9738 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:51.563508 9738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08fc8b5cf7144abeb1c0873acaea2c09: Generated new TSK 0
I20260620 12:25:51.564194 9738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:51.568935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38239
--builtin_ntp_servers=127.0.98.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 12:25:51.678368 9748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:51.678610 9748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:51.678665 9748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:51.682636 9748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:51.682744 9748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:51.682837 9748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:51.687549 9748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:51.688568 9748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:51.689985 9748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:51.699383 9753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:51.699353 9754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:51.699571 9748 server_base.cc:1034] running on GCE node
W20260620 12:25:51.699360 9756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:51.701390 9748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:51.702029 9748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:51.703188 9748 hybrid_clock.cc:648] HybridClock initialized: now 1781958351703155 us; error 49 us; skew 500 ppm
I20260620 12:25:51.705075 9748 webserver.cc:460] Webserver started at http://127.0.98.1:43005/ using document root <none> and password file <none>
I20260620 12:25:51.705662 9748 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:51.705725 9748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:51.705957 9748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:51.708024 9748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "2273e34e3c5c41c5bdf989844b36e29d"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "661aa67391aa63b6242056698b686f1d"
server_key_iv: "ed8406d5e93fd2b103282061af9eaa01"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.708575 9748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2273e34e3c5c41c5bdf989844b36e29d"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "661aa67391aa63b6242056698b686f1d"
server_key_iv: "ed8406d5e93fd2b103282061af9eaa01"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.712234 9748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:51.714640 9762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:51.715947 9748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:51.716073 9748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "2273e34e3c5c41c5bdf989844b36e29d"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "661aa67391aa63b6242056698b686f1d"
server_key_iv: "ed8406d5e93fd2b103282061af9eaa01"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.716190 9748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:51.726337 9748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:51.727033 9748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:51.727218 9748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:51.727759 9748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:51.728636 9748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:51.728682 9748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:51.728744 9748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:51.728775 9748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:51.737048 9748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41515
I20260620 12:25:51.737087 9875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41515 every 8 connection(s)
I20260620 12:25:51.737897 9748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:51.745190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 9748
I20260620 12:25:51.745324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:51.745642 392 external_mini_cluster.cc:1428] Setting key 4c308c59bb80499c0e0a7c43a1424537
I20260620 12:25:51.746989 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:51.739678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51377 (local address 127.0.98.62:38239)
0620 12:25:51.739807 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:51.739811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:51.740819 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 12:25:51.742199 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:51.742203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:51.742263 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:51.742356 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:51.743238 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.743777 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.744900 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.745141 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.745253 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.746474 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:51.746477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:51.746479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:51.746490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:51.746516 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:51.746524 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:51.746589 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:51.746782 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 12:25:51.746831 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:25:51.747782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38239
--builtin_ntp_servers=127.0.98.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 12:25:51.748042 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:51.740067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38239 (local address 127.0.98.1:51377)
0620 12:25:51.740675 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:51.740713 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:51.741996 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:51.742537 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:51.742543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:51.742624 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:51.743101 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.743113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.743916 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.743920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.744760 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.744768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.745288 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.745292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.745478 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.746164 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:51.746166 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:51.746221 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:51.746298 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:51.746612 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:51.746620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:51.746626 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:25:51.746775 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 12:25:51.747096 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260620 12:25:51.749329 9876 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38239
I20260620 12:25:51.749571 9876 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:51.750185 9876 heartbeater.cc:511] Master 127.0.98.62:38239 requested a full tablet report, sending...
I20260620 12:25:51.751608 9688 ts_manager.cc:194] Registered new tserver with Master: 2273e34e3c5c41c5bdf989844b36e29d (127.0.98.1:41515)
I20260620 12:25:51.752796 9688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51377
W20260620 12:25:51.856591 9879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:51.856864 9879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:51.856932 9879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:51.860998 9879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:51.861069 9879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:51.861153 9879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:51.865864 9879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:51.867000 9879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:51.868307 9879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:51.877836 9884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:51.877861 9885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:51.877866 9887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:51.878111 9879 server_base.cc:1034] running on GCE node
I20260620 12:25:51.878499 9879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:51.879177 9879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:51.880389 9879 hybrid_clock.cc:648] HybridClock initialized: now 1781958351880374 us; error 30 us; skew 500 ppm
I20260620 12:25:51.882185 9879 webserver.cc:460] Webserver started at http://127.0.98.2:45873/ using document root <none> and password file <none>
I20260620 12:25:51.882869 9879 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:51.882961 9879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:51.883229 9879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:51.885372 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "ebfb2944d2a94d9abee9f7c9c51a0dc0"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "0cefe4bbc63db89b3cf701b3f10b1501"
server_key_iv: "9d9ae2bea18d9a10d381f06b6d7fd660"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.885946 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ebfb2944d2a94d9abee9f7c9c51a0dc0"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "0cefe4bbc63db89b3cf701b3f10b1501"
server_key_iv: "9d9ae2bea18d9a10d381f06b6d7fd660"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.889678 9879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:25:51.892357 9893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:51.893532 9879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:51.893666 9879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "ebfb2944d2a94d9abee9f7c9c51a0dc0"
format_stamp: "Formatted at 2026-06-20 12:25:51 on dist-test-slave-bwpp"
server_key: "0cefe4bbc63db89b3cf701b3f10b1501"
server_key_iv: "9d9ae2bea18d9a10d381f06b6d7fd660"
server_key_version: "encryptionkey@0"
I20260620 12:25:51.893778 9879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:51.903896 9879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:51.904518 9879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:51.904706 9879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:51.905253 9879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:51.906147 9879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:51.906217 9879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:51.906296 9879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:51.906340 9879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:51.914836 9879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35081
I20260620 12:25:51.914862 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35081 every 8 connection(s)
I20260620 12:25:51.915812 9879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:51.924422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 9879
I20260620 12:25:51.924520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:51.924817 392 external_mini_cluster.cc:1428] Setting key 26c5ce91ec1792b116dd2b99db213f2b
I20260620 12:25:51.925242 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:51.917804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47113 (local address 127.0.98.62:38239)
0620 12:25:51.917939 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:51.917943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:51.918953 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 12:25:51.920447 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:51.920450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:51.920520 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:51.920599 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:51.921478 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.922022 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.923171 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.923401 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.923522 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.924702 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:51.924706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:51.924708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:51.924719 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:51.924738 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:51.924746 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:51.924832 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:51.925034 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 12:25:51.925098 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:25:51.926035 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:51.918137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38239 (local address 127.0.98.2:47113)
0620 12:25:51.918812 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:51.918852 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:51.920223 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:51.920776 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:51.920783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:51.920866 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:51.921332 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.921344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.922153 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.922157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.923042 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:51.923052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:51.923551 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:51.923555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:51.923717 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:51.924376 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:51.924378 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:51.924415 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:51.924510 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:51.924835 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:51.924846 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:51.924853 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:25:51.925008 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 12:25:51.925244 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260620 12:25:51.926975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38239
--builtin_ntp_servers=127.0.98.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 12:25:51.927217 10007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38239
I20260620 12:25:51.927457 10007 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:51.927950 10007 heartbeater.cc:511] Master 127.0.98.62:38239 requested a full tablet report, sending...
I20260620 12:25:51.928957 9688 ts_manager.cc:194] Registered new tserver with Master: ebfb2944d2a94d9abee9f7c9c51a0dc0 (127.0.98.2:35081)
I20260620 12:25:51.929665 9688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47113
W20260620 12:25:52.037282 10010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:52.037503 10010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:52.037537 10010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:52.041610 10010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:52.041659 10010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:52.041708 10010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:52.046229 10010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:52.047225 10010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:52.048485 10010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:52.057876 10018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.057868 10015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:52.058176 10010 server_base.cc:1034] running on GCE node
W20260620 12:25:52.057868 10016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:52.058737 10010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:52.059491 10010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:52.060676 10010 hybrid_clock.cc:648] HybridClock initialized: now 1781958352060649 us; error 41 us; skew 500 ppm
I20260620 12:25:52.062493 10010 webserver.cc:460] Webserver started at http://127.0.98.3:35739/ using document root <none> and password file <none>
I20260620 12:25:52.063100 10010 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:52.063160 10010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:52.063374 10010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:52.065317 10010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "d5152b33675045819189422ee4e407b3"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "62b0e0ba5a6bade78d41d9557e625e76"
server_key_iv: "ffb42f617b92d4de6e80a57bb7e6f5b5"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.065850 10010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d5152b33675045819189422ee4e407b3"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "62b0e0ba5a6bade78d41d9557e625e76"
server_key_iv: "ffb42f617b92d4de6e80a57bb7e6f5b5"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.069612 10010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 12:25:52.072137 10024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.073467 10010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:52.073634 10010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "d5152b33675045819189422ee4e407b3"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "62b0e0ba5a6bade78d41d9557e625e76"
server_key_iv: "ffb42f617b92d4de6e80a57bb7e6f5b5"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.073763 10010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:52.097611 10010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:52.098387 10010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:52.098611 10010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:52.099298 10010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:52.100284 10010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:52.100338 10010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.100386 10010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:52.100414 10010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.110224 10010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43809
I20260620 12:25:52.110244 10137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43809 every 8 connection(s)
I20260620 12:25:52.111220 10010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:52.113919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10010
I20260620 12:25:52.114042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:52.114482 392 external_mini_cluster.cc:1428] Setting key 489aca90704187cda76bf37f5448745c
I20260620 12:25:52.121433 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.113084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36305 (local address 127.0.98.62:38239)
0620 12:25:52.113228 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:52.113241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:52.114448 (+ 1207us) server_negotiation.cc:407] Connection header received
0620 12:25:52.116440 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:52.116444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:52.116500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:52.116586 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:52.117656 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:52.118226 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:52.119520 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:52.119807 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:52.119888 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:52.121026 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:52.121029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:52.121032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:52.121043 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:52.121061 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:52.121068 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:52.121126 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:52.121279 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 12:25:52.121324 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:25:52.122326 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.113392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38239 (local address 127.0.98.3:36305)
0620 12:25:52.114253 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:52.114316 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:52.116238 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:52.116832 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:52.116842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:52.116945 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:52.117515 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:52.117530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:52.118357 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:52.118360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:52.119407 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:52.119419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:52.119943 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:52.119946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:52.120100 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:52.120687 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:52.120689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:52.120780 (+ 91us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:52.120872 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:52.121148 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:52.121155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:52.121159 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:25:52.121269 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 12:25:52.121491 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260620 12:25:52.123472 10138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38239
I20260620 12:25:52.123721 10138 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:52.124228 10138 heartbeater.cc:511] Master 127.0.98.62:38239 requested a full tablet report, sending...
I20260620 12:25:52.125223 9688 ts_manager.cc:194] Registered new tserver with Master: d5152b33675045819189422ee4e407b3 (127.0.98.3:43809)
I20260620 12:25:52.125782 9688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36305
I20260620 12:25:52.128414 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:25:52.131371 9727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:52.130465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.0.98.62:38239)
0620 12:25:52.130629 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:52.130632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:52.130799 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:25:52.130904 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:52.130906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:52.130941 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:52.131009 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:52.131254 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42528: BlockingRecv error: recv got EOF from 127.0.0.1:42528 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:25:52.131686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 9748
I20260620 12:25:52.138856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 9879
I20260620 12:25:52.145185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10010
I20260620 12:25:52.151554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 9657
2026-06-20T12:25:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (854 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T12:25:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:52Z Disabled control of system clock
I20260620 12:25:52.180352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46821
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 12:25:52.286242 10149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:52.286556 10149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:52.286623 10149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:52.290583 10149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:52.290699 10149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:52.290727 10149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:52.290766 10149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:52.290797 10149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:52.295430 10149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:52.296468 10149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:52.297870 10149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:52.306295 10157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.306304 10154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:52.306351 10149 server_base.cc:1034] running on GCE node
W20260620 12:25:52.306303 10155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:52.307134 10149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:52.308152 10149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:52.309336 10149 hybrid_clock.cc:648] HybridClock initialized: now 1781958352309325 us; error 26 us; skew 500 ppm
I20260620 12:25:52.311244 10149 webserver.cc:460] Webserver started at http://127.0.98.62:41597/ using document root <none> and password file <none>
I20260620 12:25:52.311810 10149 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:52.311898 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:52.312126 10149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:52.314110 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "05026f9cbf684bdd89c3016cba879aa9"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "0b2544e5bd987e19ce19d1a33a378fb3"
server_key_iv: "e4cf9db83e99a938e772e6bc059301a9"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.314723 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "05026f9cbf684bdd89c3016cba879aa9"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "0b2544e5bd987e19ce19d1a33a378fb3"
server_key_iv: "e4cf9db83e99a938e772e6bc059301a9"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.318281 10149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:25:52.320667 10163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.321863 10149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:52.322016 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "05026f9cbf684bdd89c3016cba879aa9"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "0b2544e5bd987e19ce19d1a33a378fb3"
server_key_iv: "e4cf9db83e99a938e772e6bc059301a9"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.322125 10149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:52.326536 10149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:52.327145 10149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:52.327323 10149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:52.333840 10149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46821
I20260620 12:25:52.333846 10215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46821 every 8 connection(s)
I20260620 12:25:52.334764 10149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:52.336264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10149
I20260620 12:25:52.336349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:52.336644 392 external_mini_cluster.cc:1428] Setting key 210f6ecf97b25433e433fb89101da599
I20260620 12:25:52.338078 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:52.342586 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.338019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.0.98.62:46821)
0620 12:25:52.338732 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:52.338752 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:52.338795 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:25:52.339636 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:52.339684 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:52.340103 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:52.340480 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:52.341157 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:52.341164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:52.341178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:52.341189 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:52.341251 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:52.341271 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:52.341421 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:52.341489 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:25:52.341725 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260620 12:25:52.344740 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9: Bootstrap starting.
I20260620 12:25:52.346802 10216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:52.347445 10216 log.cc:824] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:52.349171 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9: No bootstrap required, opened a new log
I20260620 12:25:52.352770 10216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05026f9cbf684bdd89c3016cba879aa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260620 12:25:52.352977 10216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:52.353042 10216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05026f9cbf684bdd89c3016cba879aa9, State: Initialized, Role: FOLLOWER
I20260620 12:25:52.353489 10216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05026f9cbf684bdd89c3016cba879aa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260620 12:25:52.353675 10216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:52.353749 10216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:52.353844 10216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:52.354602 10216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05026f9cbf684bdd89c3016cba879aa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260620 12:25:52.354939 10216 leader_election.cc:302] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05026f9cbf684bdd89c3016cba879aa9; no voters:
I20260620 12:25:52.355180 10216 leader_election.cc:288] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:52.355293 10221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:52.355531 10221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 05026f9cbf684bdd89c3016cba879aa9, State: Running, Role: LEADER
I20260620 12:25:52.355818 10221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05026f9cbf684bdd89c3016cba879aa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } }
I20260620 12:25:52.356389 10216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:52.357445 10222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05026f9cbf684bdd89c3016cba879aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05026f9cbf684bdd89c3016cba879aa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } } }
I20260620 12:25:52.357565 10222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:52.357851 10223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05026f9cbf684bdd89c3016cba879aa9. Latest consensus state: current_term: 1 leader_uuid: "05026f9cbf684bdd89c3016cba879aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05026f9cbf684bdd89c3016cba879aa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46821 } } }
I20260620 12:25:52.357951 10223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:52.358049 10230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:52.361066 10230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:52.366546 10230 catalog_manager.cc:1347] Generated new cluster ID: d51e557f8c214c31b78e95c39413e756
I20260620 12:25:52.366618 10230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:52.397040 10230 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:52.397450 10230 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:52.404417 10230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05026f9cbf684bdd89c3016cba879aa9: Generated new TSK 0
I20260620 12:25:52.405193 10230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:25:52.416527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 12:25:52.523408 10240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:52.523666 10240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:52.523753 10240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:52.527870 10240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:52.527942 10240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:52.528028 10240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:52.532666 10240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:52.533689 10240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:52.535084 10240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:52.544392 10245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.544392 10248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.544432 10246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:52.545048 10240 server_base.cc:1034] running on GCE node
I20260620 12:25:52.545430 10240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:52.546078 10240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:52.547255 10240 hybrid_clock.cc:648] HybridClock initialized: now 1781958352547235 us; error 44 us; skew 500 ppm
I20260620 12:25:52.549100 10240 webserver.cc:460] Webserver started at http://127.0.98.1:44559/ using document root <none> and password file <none>
I20260620 12:25:52.549671 10240 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:52.549763 10240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:52.549971 10240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:52.552103 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "07235c2cf36142029bbc56a36a8c9fc5"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "c2ec88ed009f37aa655c607d0b41fb58"
server_key_iv: "d147da270be42cf686befb581ca60869"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.552661 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "07235c2cf36142029bbc56a36a8c9fc5"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "c2ec88ed009f37aa655c607d0b41fb58"
server_key_iv: "d147da270be42cf686befb581ca60869"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.556236 10240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:52.558688 10254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.559860 10240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:52.559989 10240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "07235c2cf36142029bbc56a36a8c9fc5"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "c2ec88ed009f37aa655c607d0b41fb58"
server_key_iv: "d147da270be42cf686befb581ca60869"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.560108 10240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:52.564554 10240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:52.565127 10240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:52.565306 10240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:52.565824 10240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:52.566710 10240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:52.566784 10240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.566848 10240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:52.566895 10240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.576048 10240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42907
I20260620 12:25:52.576071 10367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42907 every 8 connection(s)
I20260620 12:25:52.577005 10240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:52.582439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10240
I20260620 12:25:52.582561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:52.582906 392 external_mini_cluster.cc:1428] Setting key e8c6a2c72ab51d804f764a57216bd172
I20260620 12:25:52.582930 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.578802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60061 (local address 127.0.98.62:46821)
0620 12:25:52.578960 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:52.578965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:52.580046 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 12:25:52.581261 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:52.581263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:52.581298 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:52.581369 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:52.582369 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:52.582374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:52.582378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:52.582391 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:52.582419 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:52.582432 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:52.582549 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:52.582751 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 12:25:52.582798 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:25:52.583724 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.579263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.1:60061)
0620 12:25:52.579896 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:52.579938 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:52.581080 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:52.581500 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:52.581506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:52.581591 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:52.582094 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:52.582095 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:52.582133 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:52.582214 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:52.582554 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:52.582562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:52.582569 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:25:52.582786 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 12:25:52.583010 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260620 12:25:52.584810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 12:25:52.584955 10368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46821
I20260620 12:25:52.585175 10368 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:52.585619 10368 heartbeater.cc:511] Master 127.0.98.62:46821 requested a full tablet report, sending...
I20260620 12:25:52.587342 10180 ts_manager.cc:194] Registered new tserver with Master: 07235c2cf36142029bbc56a36a8c9fc5 (127.0.98.1:42907)
W20260620 12:25:52.693393 10371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:52.693655 10371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:52.693693 10371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:52.697549 10371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:52.697595 10371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:52.697649 10371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:52.702167 10371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:52.703229 10371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:52.704486 10371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:52.713691 10377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.713754 10379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.713754 10376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:52.714025 10371 server_base.cc:1034] running on GCE node
I20260620 12:25:52.714351 10371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:52.714939 10371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:52.716138 10371 hybrid_clock.cc:648] HybridClock initialized: now 1781958352716107 us; error 45 us; skew 500 ppm
I20260620 12:25:52.717820 10371 webserver.cc:460] Webserver started at http://127.0.98.2:46317/ using document root <none> and password file <none>
I20260620 12:25:52.718385 10371 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:52.718477 10371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:52.718752 10371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:52.720713 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee14f356e2d54d8396d6d268c9b756e8"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "e1eb660f119e881c848e1cbb305996ff"
server_key_iv: "2592d93a29009dcf5bc673761b61beea"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.721263 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee14f356e2d54d8396d6d268c9b756e8"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "e1eb660f119e881c848e1cbb305996ff"
server_key_iv: "2592d93a29009dcf5bc673761b61beea"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.724721 10371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:25:52.727043 10385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.728207 10371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:52.728332 10371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "ee14f356e2d54d8396d6d268c9b756e8"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "e1eb660f119e881c848e1cbb305996ff"
server_key_iv: "2592d93a29009dcf5bc673761b61beea"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.728436 10371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:52.732877 10371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:52.733438 10371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:52.733616 10371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:52.734126 10371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:52.734982 10371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:52.735056 10371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.735126 10371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:52.735175 10371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.743719 10371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39851
I20260620 12:25:52.743741 10498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39851 every 8 connection(s)
I20260620 12:25:52.744570 10371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:52.750417 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.746311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37329 (local address 127.0.98.62:46821)
0620 12:25:52.746481 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:52.746486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:52.747527 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 12:25:52.748760 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:52.748763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:52.748798 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:52.748881 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:52.749978 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:52.749981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:52.749983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:52.749995 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:52.750030 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:52.750038 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:52.750121 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:52.750236 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 12:25:52.750280 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:25:52.751318 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.746765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.2:37329)
0620 12:25:52.747377 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:52.747416 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:52.748553 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:52.749036 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:52.749042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:52.749125 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:52.749730 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:52.749731 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:52.749768 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:52.749838 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:52.750122 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:52.750128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:52.750131 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:25:52.750253 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 12:25:52.750428 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260620 12:25:52.751331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10371
I20260620 12:25:52.751453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:52.751770 392 external_mini_cluster.cc:1428] Setting key cbc14c253bb4a236aea436911a73bcd5
I20260620 12:25:52.752409 10499 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46821
I20260620 12:25:52.752647 10499 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:52.753082 10499 heartbeater.cc:511] Master 127.0.98.62:46821 requested a full tablet report, sending...
I20260620 12:25:52.753840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 12:25:52.754101 10180 ts_manager.cc:194] Registered new tserver with Master: ee14f356e2d54d8396d6d268c9b756e8 (127.0.98.2:39851)
W20260620 12:25:52.862217 10502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:52.862483 10502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:52.862555 10502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:52.866627 10502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:52.866744 10502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:52.866835 10502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:52.871488 10502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:52.872519 10502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:52.873857 10502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:52.882956 10507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.883018 10508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:52.882990 10510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:52.883214 10502 server_base.cc:1034] running on GCE node
I20260620 12:25:52.883630 10502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:52.884238 10502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:52.885417 10502 hybrid_clock.cc:648] HybridClock initialized: now 1781958352885404 us; error 27 us; skew 500 ppm
I20260620 12:25:52.887303 10502 webserver.cc:460] Webserver started at http://127.0.98.3:34975/ using document root <none> and password file <none>
I20260620 12:25:52.887912 10502 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:52.888010 10502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:52.888236 10502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:52.890264 10502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "887e680f7ff1494db9c55a3c1c17c95e"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "579d08621a4be759a8979e4df3084444"
server_key_iv: "5450508a9215b65cfd93a90e824d50fe"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.890863 10502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "887e680f7ff1494db9c55a3c1c17c95e"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "579d08621a4be759a8979e4df3084444"
server_key_iv: "5450508a9215b65cfd93a90e824d50fe"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.894461 10502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 12:25:52.896920 10516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.898175 10502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:25:52.898304 10502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "887e680f7ff1494db9c55a3c1c17c95e"
format_stamp: "Formatted at 2026-06-20 12:25:52 on dist-test-slave-bwpp"
server_key: "579d08621a4be759a8979e4df3084444"
server_key_iv: "5450508a9215b65cfd93a90e824d50fe"
server_key_version: "encryptionkey@0"
I20260620 12:25:52.898414 10502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:52.902829 10502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:52.903457 10502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:52.903638 10502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:52.904307 10502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:52.905194 10502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:52.905265 10502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.905325 10502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:52.905375 10502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:52.914119 10502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45585
I20260620 12:25:52.914141 10629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45585 every 8 connection(s)
I20260620 12:25:52.915014 10502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:52.919929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10502
I20260620 12:25:52.920061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:52.920392 392 external_mini_cluster.cc:1428] Setting key 7db722483061cd7382bdb467d9226e6e
I20260620 12:25:52.920990 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.916744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48411 (local address 127.0.98.62:46821)
0620 12:25:52.916915 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:52.916919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:52.917878 (+ 959us) server_negotiation.cc:407] Connection header received
0620 12:25:52.919193 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:52.919196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:52.919232 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:52.919304 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:52.920488 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:52.920494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:52.920497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:25:52.920514 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:52.920538 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:25:52.920547 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:52.920622 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:52.920813 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 12:25:52.920858 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:25:52.922021 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:52.917130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46821 (local address 127.0.98.3:48411)
0620 12:25:52.917751 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:52.917791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:52.919011 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:52.919453 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:52.919459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:52.919541 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:52.920175 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:25:52.920176 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:52.920232 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:25:52.920319 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:52.920636 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:52.920644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:52.920650 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:25:52.920841 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 12:25:52.921072 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260620 12:25:52.923050 10630 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46821
I20260620 12:25:52.923244 10630 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:52.923576 10630 heartbeater.cc:511] Master 127.0.98.62:46821 requested a full tablet report, sending...
I20260620 12:25:52.924450 10180 ts_manager.cc:194] Registered new tserver with Master: 887e680f7ff1494db9c55a3c1c17c95e (127.0.98.3:45585)
I20260620 12:25:52.934077 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:25:52.936895 10219 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:25:52.935992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.0.98.62:46821)
0620 12:25:52.936126 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:52.936131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:52.936281 (+ 150us) server_negotiation.cc:407] Connection header received
0620 12:25:52.936423 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:52.936426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:52.936459 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:52.936557 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:52.936775 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48988: BlockingRecv error: recv got EOF from 127.0.0.1:48988 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:25:52.937031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10240
I20260620 12:25:52.942516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10371
I20260620 12:25:52.947512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10502
I20260620 12:25:52.952598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10149
2026-06-20T12:25:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (800 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:53 dist-test-slave-bwpp krb5kdc[10640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:53 dist-test-slave-bwpp krb5kdc[10640](info): set up 2 sockets
Jun 20 12:25:53 dist-test-slave-bwpp krb5kdc[10640](info): commencing operation
krb5kdc: starting...
W20260620 12:25:53.987663 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:25:54.107127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:54.223297 10653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:54.223533 10653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:54.223572 10653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:54.227460 10653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:25:54.227504 10653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:54.227523 10653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:54.227538 10653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:25:54.227553 10653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:25:54.232554 10653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:54.233449 10653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:54.234362 10653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:54.242537 10658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:54.242523 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:54.242544 10661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:54.242746 10653 server_base.cc:1034] running on GCE node
I20260620 12:25:54.243458 10653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:54.244511 10653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:54.245692 10653 hybrid_clock.cc:648] HybridClock initialized: now 1781958354245670 us; error 56 us; skew 500 ppm
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:54.251648 10653 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:25:54.252545 10653 webserver.cc:460] Webserver started at http://127.0.98.62:38687/ using document root <none> and password file <none>
I20260620 12:25:54.253083 10653 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:54.253170 10653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:54.253400 10653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:54.255549 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "4f9e07f09d271e80fabc1141881543c7"
server_key_iv: "a92c6eb1e47377a1b5c532c6614c514f"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.256105 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "4f9e07f09d271e80fabc1141881543c7"
server_key_iv: "a92c6eb1e47377a1b5c532c6614c514f"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.259730 10653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:25:54.261986 10668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.263265 10653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:54.263427 10653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "4f9e07f09d271e80fabc1141881543c7"
server_key_iv: "a92c6eb1e47377a1b5c532c6614c514f"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.263533 10653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:54.284590 10653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:54.287432 10653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:54.287638 10653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:54.294593 10653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39003
I20260620 12:25:54.294889 10720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39003 every 8 connection(s)
I20260620 12:25:54.295568 10653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:25:54.298161 10721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:54.302924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10653
I20260620 12:25:54.303057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:25:54.303346 392 external_mini_cluster.cc:1428] Setting key 65b42ddab70d34aad0963b6ba23f69ed
I20260620 12:25:54.303583 10721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc: Bootstrap starting.
I20260620 12:25:54.305923 10721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:54.306843 10721 log.cc:824] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:54.308811 10721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc: No bootstrap required, opened a new log
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:54.313118 10721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260620 12:25:54.313416 10721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:54.313506 10721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ce7ebd6163402a8aad1ea56e4ec2dc, State: Initialized, Role: FOLLOWER
I20260620 12:25:54.314006 10721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260620 12:25:54.314175 10721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:25:54.314268 10721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:25:54.314365 10721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:54.315255 10721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260620 12:25:54.315654 10721 leader_election.cc:302] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ce7ebd6163402a8aad1ea56e4ec2dc; no voters:
I20260620 12:25:54.315985 10721 leader_election.cc:288] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:25:54.316082 10726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:54.316274 10726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [term 1 LEADER]: Becoming Leader. State: Replica: 75ce7ebd6163402a8aad1ea56e4ec2dc, State: Running, Role: LEADER
I20260620 12:25:54.316574 10726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } }
I20260620 12:25:54.317224 10721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:25:54.318361 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:54.304584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.98.62:39003)
0620 12:25:54.304963 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:54.304979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:54.305013 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:25:54.305598 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:54.305612 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:54.305919 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:54.306231 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:54.307137 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.308290 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.309419 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.309864 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.309977 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:54.312754 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:54.312759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:54.312771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:54.312796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:54.314799 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.315215 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.315220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.315226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.315352 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.315671 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.315673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.315674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.316197 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:54.316408 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:54.316980 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 12:25:54.317284 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":131,"threads_started":1}
I20260620 12:25:54.319999 10728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ce7ebd6163402a8aad1ea56e4ec2dc. Latest consensus state: current_term: 1 leader_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } } }
I20260620 12:25:54.320113 10728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [sys.catalog]: This master's current role is: LEADER
I20260620 12:25:54.320318 10727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ce7ebd6163402a8aad1ea56e4ec2dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39003 } } }
I20260620 12:25:54.320395 10727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc [sys.catalog]: This master's current role is: LEADER
W20260620 12:25:54.322273 10741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:25:54.322367 10741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:25:54.322441 10742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:25:54.323166 10742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:25:54.328450 10742 catalog_manager.cc:1347] Generated new cluster ID: c5240a146ed241e888878fdb9fd62283
I20260620 12:25:54.328505 10742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:25:54.339334 10742 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:25:54.340480 10742 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:25:54.357879 10742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc: Generated new TSK 0
I20260620 12:25:54.358758 10742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:25:54.386468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39003
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:54.494472 10747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:54.494802 10747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:54.494882 10747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:54.498795 10747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:54.498871 10747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:54.498975 10747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:25:54.504118 10747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:54.505106 10747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:54.506102 10747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:54.515267 10753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:54.515275 10755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:54.515399 10752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:54.515789 10747 server_base.cc:1034] running on GCE node
I20260620 12:25:54.516186 10747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:54.516810 10747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:54.517964 10747 hybrid_clock.cc:648] HybridClock initialized: now 1781958354517939 us; error 37 us; skew 500 ppm
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:54.520895 10747 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:25:54.521734 10747 webserver.cc:460] Webserver started at http://127.0.98.1:38089/ using document root <none> and password file <none>
I20260620 12:25:54.522295 10747 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:54.522356 10747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:54.522527 10747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:54.524524 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ca524d17f34408285475ba2bf46491f"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "8a71d6b5f0deaafbc848f9ff04001e47"
server_key_iv: "a1b7e12ee97fd361b977e192085622cf"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.525029 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ca524d17f34408285475ba2bf46491f"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "8a71d6b5f0deaafbc848f9ff04001e47"
server_key_iv: "a1b7e12ee97fd361b977e192085622cf"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.528643 10747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 12:25:54.531029 10762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.532267 10747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:25:54.532377 10747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "7ca524d17f34408285475ba2bf46491f"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "8a71d6b5f0deaafbc848f9ff04001e47"
server_key_iv: "a1b7e12ee97fd361b977e192085622cf"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.532480 10747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:54.547012 10747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:54.550133 10747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:54.550330 10747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:54.550946 10747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:54.551713 10747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:54.551759 10747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.551796 10747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:54.551851 10747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.560616 10747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36317
I20260620 12:25:54.560637 10875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36317 every 8 connection(s)
I20260620 12:25:54.561447 10747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:25:54.562611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10747
I20260620 12:25:54.562762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:25:54.563062 392 external_mini_cluster.cc:1428] Setting key a05bfc9fdaf480d1e262d3d52e2a346d
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:25:54.576653 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:54.563272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33397 (local address 127.0.98.62:39003)
0620 12:25:54.563517 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:54.563522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:54.564521 (+ 999us) server_negotiation.cc:407] Connection header received
0620 12:25:54.565781 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:54.565787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:54.565849 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:54.565945 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:54.567580 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.568469 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.569633 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.570095 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.570230 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:54.573055 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:54.573058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:54.573060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:54.573084 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:54.574466 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.575060 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.575063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.575064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.575138 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.575479 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.575481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.575483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.575660 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:54.575771 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:54.576345 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 12:25:54.576468 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260620 12:25:54.577620 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:54.563752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39003 (local address 127.0.98.1:33397)
0620 12:25:54.564252 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:54.564416 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:54.565572 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:54.566159 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:54.566170 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:54.566733 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:54.567423 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:54.567445 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.568598 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.568601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:54.569480 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:54.569490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.570244 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.570247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:54.570391 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:54.571053 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:54.571069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:54.572857 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:54.574598 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:54.574604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:54.574616 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:54.574965 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:54.575228 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:54.575231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:54.575232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:54.575384 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:54.575773 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:54.575780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:54.576102 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 12:25:54.576291 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:25:54.576623 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":130,"threads_started":1}
I20260620 12:25:54.579010 10876 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39003
I20260620 12:25:54.579265 10876 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:54.579754 10876 heartbeater.cc:511] Master 127.0.98.62:39003 requested a full tablet report, sending...
I20260620 12:25:54.581276 10685 ts_manager.cc:194] Registered new tserver with Master: 7ca524d17f34408285475ba2bf46491f (127.0.98.1:36317)
I20260620 12:25:54.582415 10685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:25:54.591205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39003
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:54.699928 10881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:54.700160 10881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:54.700199 10881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:54.704119 10881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:54.704174 10881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:54.704228 10881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:25:54.709304 10881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:54.710224 10881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:54.711189 10881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:54.720587 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:54.720732 10881 server_base.cc:1034] running on GCE node
W20260620 12:25:54.720616 10886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:54.720947 10887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:54.721413 10881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:54.722018 10881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:54.723179 10881 hybrid_clock.cc:648] HybridClock initialized: now 1781958354723167 us; error 24 us; skew 500 ppm
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:54.725821 10881 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:25:54.726720 10881 webserver.cc:460] Webserver started at http://127.0.98.2:46515/ using document root <none> and password file <none>
I20260620 12:25:54.727257 10881 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:54.727380 10881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:54.727604 10881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:54.729640 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c109ca9497347b8bcc1d917ccaad90b"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "d0ff31bec330e3dc901ea1c1497de94b"
server_key_iv: "a33e0fc94c142c07e066f760e2539b95"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.730194 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c109ca9497347b8bcc1d917ccaad90b"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "d0ff31bec330e3dc901ea1c1497de94b"
server_key_iv: "a33e0fc94c142c07e066f760e2539b95"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.733784 10881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:25:54.736133 10896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.737391 10881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:54.737514 10881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "2c109ca9497347b8bcc1d917ccaad90b"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "d0ff31bec330e3dc901ea1c1497de94b"
server_key_iv: "a33e0fc94c142c07e066f760e2539b95"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.737607 10881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:54.754077 10881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:54.757064 10881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:54.757269 10881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:54.757834 10881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:54.758705 10881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:54.758762 10881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.758801 10881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:54.758816 10881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.767917 10881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44285
I20260620 12:25:54.767962 11009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44285 every 8 connection(s)
I20260620 12:25:54.768792 10881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:25:54.777069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 10881
I20260620 12:25:54.777200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:25:54.777542 392 external_mini_cluster.cc:1428] Setting key fad51b94e91ac9f6ba348beb6357c361
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:54.783125 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:54.770536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57841 (local address 127.0.98.62:39003)
0620 12:25:54.770728 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:54.770732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:54.771556 (+ 824us) server_negotiation.cc:407] Connection header received
0620 12:25:54.772837 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:54.772840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:54.772919 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:54.773041 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:54.774159 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.774772 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.775915 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.776153 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.776264 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:54.779285 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:54.779288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:54.779290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:54.779311 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:54.780749 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.781370 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.781373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.781374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.781460 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.781890 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.781896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.781901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.782143 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:54.782257 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:54.782750 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 12:25:54.782885 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:54.783836 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:54.770986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39003 (local address 127.0.98.2:57841)
0620 12:25:54.771402 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:54.771443 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:54.772606 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:54.773207 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:54.773213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:54.773558 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:54.774015 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:54.774028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.774897 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.774904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:54.775770 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:54.775780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.776300 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.776304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:54.776468 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:54.777057 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:54.777078 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:54.779119 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:54.780863 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:54.780871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:54.780887 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:54.781249 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:54.781570 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:54.781573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:54.781575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:54.781762 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:54.782248 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:54.782253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:54.782476 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 12:25:54.782644 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 12:25:54.782954 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260620 12:25:54.785022 11010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39003
I20260620 12:25:54.785244 11010 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:54.785709 11010 heartbeater.cc:511] Master 127.0.98.62:39003 requested a full tablet report, sending...
I20260620 12:25:54.786777 10685 ts_manager.cc:194] Registered new tserver with Master: 2c109ca9497347b8bcc1d917ccaad90b (127.0.98.2:44285)
I20260620 12:25:54.787338 10685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57841
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:25:54.804008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39003
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:25:54.911563 11015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:25:54.911796 11015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:25:54.911836 11015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:25:54.915776 11015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:25:54.915825 11015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:25:54.915908 11015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:25:54.921180 11015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:25:54.922127 11015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:25:54.923111 11015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:25:54.932435 11023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:54.932487 11015 server_base.cc:1034] running on GCE node
W20260620 12:25:54.932446 11020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:25:54.932435 11021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:25:54.933167 11015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:25:54.933794 11015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:25:54.934978 11015 hybrid_clock.cc:648] HybridClock initialized: now 1781958354934971 us; error 30 us; skew 500 ppm
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:25:54.937777 11015 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:25:54.938697 11015 webserver.cc:460] Webserver started at http://127.0.98.3:33941/ using document root <none> and password file <none>
I20260620 12:25:54.939229 11015 fs_manager.cc:359] Metadata directory not provided
I20260620 12:25:54.939327 11015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:25:54.939546 11015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:25:54.941481 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "8968e46f3c7202eb0a960e97fc260dd9"
server_key_iv: "b21d0d7dcbf6c712797c57b7da6fbad9"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.942015 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "8968e46f3c7202eb0a960e97fc260dd9"
server_key_iv: "b21d0d7dcbf6c712797c57b7da6fbad9"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.945542 11015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:25:54.947932 11030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.949086 11015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:25:54.949213 11015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79"
format_stamp: "Formatted at 2026-06-20 12:25:54 on dist-test-slave-bwpp"
server_key: "8968e46f3c7202eb0a960e97fc260dd9"
server_key_iv: "b21d0d7dcbf6c712797c57b7da6fbad9"
server_key_version: "encryptionkey@0"
I20260620 12:25:54.949352 11015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:25:54.967680 11015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:25:54.970942 11015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:25:54.971138 11015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:25:54.971714 11015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:25:54.972545 11015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:25:54.972609 11015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.972672 11015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:25:54.972723 11015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:25:54.981609 11015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38653
I20260620 12:25:54.981637 11143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38653 every 8 connection(s)
I20260620 12:25:54.982452 11015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:25:54.989706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 11015
I20260620 12:25:54.989817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:25:54.990164 392 external_mini_cluster.cc:1428] Setting key a342ce45165828c120bc24bdd60c27f3
Jun 20 12:25:54 dist-test-slave-bwpp krb5kdc[10640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:25:54.996274 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:54.984200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37921 (local address 127.0.98.62:39003)
0620 12:25:54.984349 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:54.984354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:54.985275 (+ 921us) server_negotiation.cc:407] Connection header received
0620 12:25:54.986499 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:54.986502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:54.986546 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:54.986626 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:54.987959 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.988489 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.989702 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.990020 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.990109 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:54.992794 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:54.992797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:54.992800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:54.992826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:54.994260 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.994817 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.994822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.994824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.994903 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:54.995238 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:54.995241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:54.995243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:54.995422 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:54.995527 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:54.996001 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 12:25:54.996111 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:25:54.997066 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:54.984701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39003 (local address 127.0.98.3:37921)
0620 12:25:54.985124 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:54.985165 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:54.986303 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:54.986911 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:54.986918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:54.987312 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:25:54.987810 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:54.987824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.988607 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.988610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:54.989571 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:54.989584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:54.990172 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:54.990180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:54.990380 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:54.991066 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:25:54.991081 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:25:54.992630 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:25:54.994398 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:54.994403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:54.994416 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:54.994715 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:54.994999 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:25:54.995002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:25:54.995004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:25:54.995137 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:25:54.995525 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:25:54.995532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:25:54.995762 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 12:25:54.995949 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 12:25:54.996247 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260620 12:25:54.998037 11144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39003
I20260620 12:25:54.998255 11144 heartbeater.cc:464] Registering TS with master...
I20260620 12:25:54.998790 11144 heartbeater.cc:511] Master 127.0.98.62:39003 requested a full tablet report, sending...
I20260620 12:25:54.999693 10685 ts_manager.cc:194] Registered new tserver with Master: e2c125a84eea4b99a5b6d15cc2ee4d79 (127.0.98.3:38653)
I20260620 12:25:55.000247 10685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37921
I20260620 12:25:55.003829 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:25:55.012354 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.005739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.0.98.62:39003)
0620 12:25:55.005860 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:55.005863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:55.006078 (+ 215us) server_negotiation.cc:407] Connection header received
0620 12:25:55.006160 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:55.006162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:55.006196 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:55.006262 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:25:55.006971 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.007452 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.008225 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.008421 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.008492 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.009744 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:25:55.009747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:25:55.009748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:25:55.009763 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:25:55.011050 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:55.011411 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:55.011414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:55.011415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:55.011479 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:25:55.011690 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:25:55.011692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:25:55.011693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:25:55.011873 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:25:55.011969 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:55.012167 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 12:25:55.012249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:25:55.014961 10685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:25:55.018047 10685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:25:55.030872 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.025900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44285 (local address 127.0.0.1:39104)
0620 12:25:55.026376 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:55.026421 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:55.026538 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:55.027152 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:55.027157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:55.027181 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:55.027313 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.027320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.028579 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.028585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.029445 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.029453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.030307 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.030311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.030446 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.030485 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:25:55.030582 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 12:25:55.030718 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260620 12:25:55.031021 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.026058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.0.98.2:44285)
0620 12:25:55.026599 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:55.026614 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:55.026638 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:25:55.026741 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:55.026747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:55.026959 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:55.027144 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:55.027468 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.028465 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.029565 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.030184 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.030307 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.030602 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:55.030685 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 12:25:55.030768 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":105,"threads_started":1}
I20260620 12:25:55.032810 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.026057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36317 (local address 127.0.0.1:49348)
0620 12:25:55.026781 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:55.026795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:55.026875 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:55.027960 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:55.027963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:55.027980 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:55.028076 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.028083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.029056 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.029062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.030284 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.030294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.032446 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.032450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.032536 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.032557 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:25:55.032622 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:25:55.032688 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":95,"threads_started":1}
I20260620 12:25:55.032853 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.026697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.0.98.1:36317)
0620 12:25:55.027489 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:55.027518 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:55.027540 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:25:55.027616 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:55.027621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:55.027814 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:55.027983 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:55.028205 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.028920 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.030468 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.032318 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.032446 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.032587 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:55.032642 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 12:25:55.032721 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":98,"threads_started":1}
I20260620 12:25:55.034027 10944 tablet_service.cc:1467] Processing CreateTablet for tablet 8c579cc63a7a4180812666ec7cad1f2a (DEFAULT_TABLE table=test-table [id=053dc54c52ad4d33a6e7f57f383eb186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:55.035174 10944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c579cc63a7a4180812666ec7cad1f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:55.035457 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.026558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38653 (local address 127.0.0.1:51460)
0620 12:25:55.027034 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:55.027050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:55.027121 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:55.027967 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:55.027970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:55.027995 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:55.028084 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.028089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.030404 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.030407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.033840 (+ 3433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.033848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.035020 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.035024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.035115 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.035134 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 12:25:55.035291 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 12:25:55.035361 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":2529,"spinlock_wait_cycles":1792,"thread_start_us":109,"threads_started":1}
I20260620 12:25:55.035939 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.026979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.98.3:38653)
0620 12:25:55.027390 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:55.027403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:55.027426 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:25:55.027492 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:55.027497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:55.027675 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:55.027810 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:55.029098 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.030250 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.033940 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.034843 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.035029 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.035157 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:55.035677 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 12:25:55.035823 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":151,"threads_started":1}
I20260620 12:25:55.036854 10810 tablet_service.cc:1467] Processing CreateTablet for tablet 8c579cc63a7a4180812666ec7cad1f2a (DEFAULT_TABLE table=test-table [id=053dc54c52ad4d33a6e7f57f383eb186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:55.037804 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c579cc63a7a4180812666ec7cad1f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:55.038414 11078 tablet_service.cc:1467] Processing CreateTablet for tablet 8c579cc63a7a4180812666ec7cad1f2a (DEFAULT_TABLE table=test-table [id=053dc54c52ad4d33a6e7f57f383eb186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:25:55.039251 11078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c579cc63a7a4180812666ec7cad1f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:25:55.043090 11161 tablet_bootstrap.cc:492] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: Bootstrap starting.
I20260620 12:25:55.044582 11162 tablet_bootstrap.cc:492] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f: Bootstrap starting.
I20260620 12:25:55.045459 11163 tablet_bootstrap.cc:492] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Bootstrap starting.
I20260620 12:25:55.045933 11161 tablet_bootstrap.cc:654] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:55.046805 11162 tablet_bootstrap.cc:654] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:55.046836 11161 log.cc:824] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:55.047501 11163 tablet_bootstrap.cc:654] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Neither blocks nor log segments found. Creating new log.
I20260620 12:25:55.048009 11162 log.cc:824] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:55.048602 11163 log.cc:824] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Log is configured to *not* fsync() on all Append() calls
I20260620 12:25:55.048909 11161 tablet_bootstrap.cc:492] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: No bootstrap required, opened a new log
I20260620 12:25:55.049089 11161 ts_tablet_manager.cc:1397] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:25:55.049829 11162 tablet_bootstrap.cc:492] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f: No bootstrap required, opened a new log
I20260620 12:25:55.050026 11162 ts_tablet_manager.cc:1397] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 12:25:55.050320 11163 tablet_bootstrap.cc:492] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: No bootstrap required, opened a new log
I20260620 12:25:55.050495 11163 ts_tablet_manager.cc:1397] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:25:55.052927 11161 raft_consensus.cc:348] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.053195 11161 raft_consensus.cc:374] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:55.053257 11161 raft_consensus.cc:729] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c109ca9497347b8bcc1d917ccaad90b, State: Initialized, Role: FOLLOWER
I20260620 12:25:55.053328 11163 raft_consensus.cc:348] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.053589 11163 raft_consensus.cc:374] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:55.053655 11163 raft_consensus.cc:729] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c125a84eea4b99a5b6d15cc2ee4d79, State: Initialized, Role: FOLLOWER
I20260620 12:25:55.053700 11161 consensus_queue.cc:260] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.053808 11162 raft_consensus.cc:348] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.054109 11162 raft_consensus.cc:374] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:25:55.054075 11163 consensus_queue.cc:260] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.054174 11162 raft_consensus.cc:729] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca524d17f34408285475ba2bf46491f, State: Initialized, Role: FOLLOWER
I20260620 12:25:55.054462 11010 heartbeater.cc:503] Master 127.0.98.62:39003 was elected leader, sending a full tablet report...
I20260620 12:25:55.054549 11162 consensus_queue.cc:260] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.055049 11163 ts_tablet_manager.cc:1428] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260620 12:25:55.055320 11144 heartbeater.cc:503] Master 127.0.98.62:39003 was elected leader, sending a full tablet report...
I20260620 12:25:55.055401 11161 ts_tablet_manager.cc:1428] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:25:55.055409 11162 ts_tablet_manager.cc:1428] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 12:25:55.055687 10876 heartbeater.cc:503] Master 127.0.98.62:39003 was elected leader, sending a full tablet report...
W20260620 12:25:55.062583 10877 tablet.cc:2369] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:25:55.071429 11168 raft_consensus.cc:482] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:25:55.071655 11168 raft_consensus.cc:504] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.073221 11168 leader_election.cc:288] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c109ca9497347b8bcc1d917ccaad90b (127.0.98.2:44285), 7ca524d17f34408285475ba2bf46491f (127.0.98.1:36317)
I20260620 12:25:55.078045 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.073976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39951 (local address 127.0.98.2:44285)
0620 12:25:55.074109 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:55.074114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:55.074128 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:55.074182 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:55.074184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:55.074223 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:55.074318 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:55.074599 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.075460 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.076740 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.077461 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.077569 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.077633 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:55.077854 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 12:25:55.077926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:25:55.078045 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.073394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44285 (local address 127.0.98.3:39951)
0620 12:25:55.073675 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:55.073691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:55.073780 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:55.074354 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:55.074357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:55.074373 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:55.074489 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.074494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.075590 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.075594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.076612 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.076621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.077575 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.077579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.077683 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.077704 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:25:55.077827 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 12:25:55.077904 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":1}
I20260620 12:25:55.078967 10964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579cc63a7a4180812666ec7cad1f2a" candidate_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c109ca9497347b8bcc1d917ccaad90b" is_pre_election: true
I20260620 12:25:55.079066 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.073451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36317 (local address 127.0.98.3:38379)
0620 12:25:55.073963 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:55.073979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:25:55.074071 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:25:55.074617 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:25:55.074620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:25:55.074635 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:25:55.074752 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.074758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.076028 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.076035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.077373 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:25:55.077387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.078589 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.078594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:25:55.078744 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.078774 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:25:55.078855 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:25:55.078929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":108,"threads_started":1}
I20260620 12:25:55.079080 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.073990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38379 (local address 127.0.98.1:36317)
0620 12:25:55.074118 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:55.074124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:55.074139 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:25:55.074192 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:55.074194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:55.074239 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:55.074326 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:25:55.074976 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.075896 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.077553 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.078293 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.078383 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.078464 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:55.078885 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 12:25:55.078973 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:25:55.079294 10964 raft_consensus.cc:2454] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c125a84eea4b99a5b6d15cc2ee4d79 in term 0.
I20260620 12:25:55.079764 10830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579cc63a7a4180812666ec7cad1f2a" candidate_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ca524d17f34408285475ba2bf46491f" is_pre_election: true
I20260620 12:25:55.079836 11032 leader_election.cc:302] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c109ca9497347b8bcc1d917ccaad90b, e2c125a84eea4b99a5b6d15cc2ee4d79; no voters:
I20260620 12:25:55.080050 10830 raft_consensus.cc:2454] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c125a84eea4b99a5b6d15cc2ee4d79 in term 0.
I20260620 12:25:55.080145 11168 raft_consensus.cc:2781] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:25:55.080230 11168 raft_consensus.cc:482] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:25:55.080271 11168 raft_consensus.cc:3037] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:55.081311 11168 raft_consensus.cc:504] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.081702 11168 leader_election.cc:288] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [CANDIDATE]: Term 1 election: Requested vote from peers 2c109ca9497347b8bcc1d917ccaad90b (127.0.98.2:44285), 7ca524d17f34408285475ba2bf46491f (127.0.98.1:36317)
I20260620 12:25:55.082048 10964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579cc63a7a4180812666ec7cad1f2a" candidate_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c109ca9497347b8bcc1d917ccaad90b"
I20260620 12:25:55.082149 10830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579cc63a7a4180812666ec7cad1f2a" candidate_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ca524d17f34408285475ba2bf46491f"
I20260620 12:25:55.082194 10964 raft_consensus.cc:3037] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:55.082288 10830 raft_consensus.cc:3037] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:25:55.083400 10964 raft_consensus.cc:2454] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c125a84eea4b99a5b6d15cc2ee4d79 in term 1.
I20260620 12:25:55.083734 11032 leader_election.cc:302] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c109ca9497347b8bcc1d917ccaad90b, e2c125a84eea4b99a5b6d15cc2ee4d79; no voters:
I20260620 12:25:55.083835 10830 raft_consensus.cc:2454] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c125a84eea4b99a5b6d15cc2ee4d79 in term 1.
I20260620 12:25:55.083986 11168 raft_consensus.cc:2781] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:25:55.084245 11168 raft_consensus.cc:686] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 1 LEADER]: Becoming Leader. State: Replica: e2c125a84eea4b99a5b6d15cc2ee4d79, State: Running, Role: LEADER
I20260620 12:25:55.084609 11168 consensus_queue.cc:237] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } }
I20260620 12:25:55.088135 10685 catalog_manager.cc:5495] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c125a84eea4b99a5b6d15cc2ee4d79 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c125a84eea4b99a5b6d15cc2ee4d79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } health_report { overall_health: HEALTHY } } }
I20260620 12:25:55.111543 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.107805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.0.98.3:38653)
0620 12:25:55.107940 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:25:55.107945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:25:55.107959 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:25:55.108069 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:25:55.108072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:25:55.108139 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:25:55.108252 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:25:55.108491 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.109002 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.109936 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:25:55.110179 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:25:55.110288 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:25:55.110645 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:25:55.111142 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:25:55.111279 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:25:55.111347 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:25:55.111404 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:25:55.117861 10964 raft_consensus.cc:1261] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 1 FOLLOWER]: Refusing update from remote peer e2c125a84eea4b99a5b6d15cc2ee4d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:55.117861 10830 raft_consensus.cc:1261] T 8c579cc63a7a4180812666ec7cad1f2a P 7ca524d17f34408285475ba2bf46491f [term 1 FOLLOWER]: Refusing update from remote peer e2c125a84eea4b99a5b6d15cc2ee4d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:25:55.118667 11168 consensus_queue.cc:1035] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca524d17f34408285475ba2bf46491f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:55.118875 11171 consensus_queue.cc:1035] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c109ca9497347b8bcc1d917ccaad90b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:25:55.127931 11176 mvcc.cc:205] Tried to move back new op lower bound from 7298901422554767360 to 7298901422429622272. Current Snapshot: MvccSnapshot[applied={T|T < 7298901422554767360}]
I20260620 12:25:55.129185 11174 mvcc.cc:205] Tried to move back new op lower bound from 7298901422554767360 to 7298901422429622272. Current Snapshot: MvccSnapshot[applied={T|T < 7298901422554767360 or (T in {7298901422554767360})}]
I20260620 12:25:55.134258 10684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:55.134394 10684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:25:55.136759 10684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75ce7ebd6163402a8aad1ea56e4ec2dc: Sending DeleteTablet for 3 replicas of tablet 8c579cc63a7a4180812666ec7cad1f2a
I20260620 12:25:55.137619 10944 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c579cc63a7a4180812666ec7cad1f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39104
I20260620 12:25:55.137692 11078 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c579cc63a7a4180812666ec7cad1f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51460
I20260620 12:25:55.138139 11183 tablet_replica.cc:330] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: stopping tablet replica
I20260620 12:25:55.138126 10810 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c579cc63a7a4180812666ec7cad1f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:25:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49348
I20260620 12:25:55.138262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10747
I20260620 12:25:55.141229 11184 tablet_replica.cc:330] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: stopping tablet replica
I20260620 12:25:55.141578 11184 raft_consensus.cc:2229] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:25:55.141889 11184 raft_consensus.cc:2258] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:55.142351 11183 raft_consensus.cc:2229] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:25:55.142779 11183 raft_consensus.cc:2258] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:25:55.144047 11184 ts_tablet_manager.cc:1905] T 8c579cc63a7a4180812666ec7cad1f2a P 2c109ca9497347b8bcc1d917ccaad90b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:25:55.144207 11183 ts_tablet_manager.cc:1905] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 12:25:55.145623 10669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36317 (error 108)
I20260620 12:25:55.145694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10881
I20260620 12:25:55.147022 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.146723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36317 (local address 127.0.0.1:49356)
0620 12:25:55.146878 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:55.146939 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 12:25:55.147274 10669 catalog_manager.cc:4584] TS 7ca524d17f34408285475ba2bf46491f (127.0.98.1:36317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c579cc63a7a4180812666ec7cad1f2a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36317: connect: Connection refused (error 111)
I20260620 12:25:55.148739 11183 ts_tablet_manager.cc:1918] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:25:55.148841 11183 log.cc:1196] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/8c579cc63a7a4180812666ec7cad1f2a
I20260620 12:25:55.149185 11183 ts_tablet_manager.cc:1939] T 8c579cc63a7a4180812666ec7cad1f2a P e2c125a84eea4b99a5b6d15cc2ee4d79: Deleting consensus metadata
I20260620 12:25:55.150398 10670 catalog_manager.cc:4841] TS e2c125a84eea4b99a5b6d15cc2ee4d79 (127.0.98.3:38653): tablet 8c579cc63a7a4180812666ec7cad1f2a (table test-table [id=053dc54c52ad4d33a6e7f57f383eb186]) successfully deleted
I20260620 12:25:55.153597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 11015
I20260620 12:25:55.155153 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:25:55.153806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44285 (local address 127.0.0.1:39112)
0620 12:25:55.155021 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0620 12:25:55.155077 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1132}
I20260620 12:25:55.158793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 10653
2026-06-20T12:25:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2208 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:25:55 dist-test-slave-bwpp krb5kdc[11188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:25:55 dist-test-slave-bwpp krb5kdc[11188](info): set up 2 sockets
Jun 20 12:25:55 dist-test-slave-bwpp krb5kdc[11188](info): commencing operation
krb5kdc: starting...
W20260620 12:25:56.191874 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:25:56 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:25:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 12:25:56.351114 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiQxjdC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/postgres -l logfile start
2026-06-20 12:25:58.685 UTC [11218] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 12:25:58.685 UTC [11218] LOG: listening on IPv4 address "127.0.98.20", port 52071
2026-06-20 12:25:58.689 UTC [11218] LOG: listening on Unix socket "/tmp/.s.PGSQL.52071"
2026-06-20 12:25:58.722 UTC [11221] LOG: database system was shut down at 2026-06-20 12:25:57 UTC
2026-06-20 12:25:58.726 UTC [11218] LOG: database system is ready to accept connections
I20260620 12:25:59.667557 392 mini_postgres.cc:96] Postgres bound to 52071
2026-06-20 12:25:59.674 UTC [11229] FATAL: database "slave" does not exist
127.0.98.20:52071 - accepting connections
I20260620 12:25:59.675007 392 mini_ranger.cc:161] Starting Ranger...
I20260620 12:25:59.689779 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 12:26:00.134147 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 12:26:00.134235 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-admin
2026-06-20 12:26:00,405 [I] DB FLAVOR :POSTGRES
2026-06-20 12:26:00,405 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:26:00,405 [I] Checking connection..
2026-06-20 12:26:00,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 12:26:00,644 [I] Checking connection passed.
2026-06-20 12:26:00,644 [I] --------- Verifying version history table ---------
2026-06-20 12:26:00,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:26:00,871 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:26:00,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:26:01,113 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:26:01,113 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 12:26:01,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 12:26:01.365 UTC [11293] WARNING: there is no transaction in progress
2026-06-20 12:26:01,371 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 12:26:01,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:26:01,644 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 12:26:01,644 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:26:01,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 12:26:01,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 12:26:02,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bwpp.c.gcp-upstream.internal','N') ;"
2026-06-20 12:26:02,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:26:02,630 [I] Table x_portal_user does not exist in database ranger
2026-06-20 12:26:02,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:26:02,875 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 12:26:02,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:26:03,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:26:03,383 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 12:26:03,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 12:26:04.328 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.344 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.359 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.372 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.389 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.405 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.455 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.461 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.474 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.484 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.493 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.505 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.514 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.520 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.525 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:04.531 UTC [11401] WARNING: there is no transaction in progress
2026-06-20 12:26:05,098 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 12:26:05,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 12:26:05,363 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 12:26:05,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:26:05,619 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 12:26:05,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:26:05,871 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 12:26:05,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:26:06,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:26:06,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bwpp.c.gcp-upstream.internal';"
2026-06-20 12:26:06,610 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 12:26:06,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 12:26:06,857 [I] DB_PATCHES have already been applied
I20260620 12:26:06.863178 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiQxjdC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 12:26:06.863243 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 12:26:06.866588 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56043
Jun 20, 2026 12:26:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:26:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:26:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:26:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:26:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:26:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:26:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 12:26:07.978830 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.304s user 0.005s sys 0.005s
Jun 20, 2026 12:26:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:26:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:26:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:26:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:26:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 12:26:29.544644 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 12:26:29 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958389, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 12:26:29.621791 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 12:26:29.631202 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 12:26:30.381927 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 12:26:30.382023 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms
I20260620 12:26:30.382117 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52071
2026-06-20 12:26:30,568 [I] DB FLAVOR :POSTGRES
2026-06-20 12:26:30,568 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:26:30,568 [I] Checking connection
2026-06-20 12:26:30,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 12:26:30,799 [I] connection success
2026-06-20 12:26:30,799 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 12:26:30,800 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 12:26:30,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 12:26:31,030 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 12:26:31,031 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:26:31,031 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 12:26:31,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 12:26:31,300 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 12:26:31.759902 392 mini_ranger_kms.cc:310] Created kms service
I20260620 12:26:31.927599 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 12:26:31.971659 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 12:26:31 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958389, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:26:32.381140 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 12:26:32.381328 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 12:26:32.381356 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 12:26:32.385439 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11581
I20260620 12:26:32.385555 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:50093
Jun 20, 2026 12:26:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:26:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:26:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:26:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:26:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:26:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 12:26:34.667119 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.045s user 0.005s sys 0.165s
I20260620 12:26:34.667254 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 12:26:34.667403 392 mini_ranger_kms.cc:401] 127.0.98.20:50093/kms/v1/keys
Jun 20 12:26:34 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958394, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 12:26:35 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958394, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:26:35 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958394, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:26:37 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:26:37.887710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45235
--ranger_config_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50093/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:26:37.999138 11632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:26:37.999431 11632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:26:37.999517 11632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:26:38.003512 11632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:26:38.003589 11632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:26:38.003625 11632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:26:38.003674 11632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:26:38.003723 11632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:26:38.009159 11632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:50093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45235
--ranger_config_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:26:38.010237 11632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:26:38.011301 11632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:26:38.019974 11638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:26:38.020109 11640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:26:38.020248 11632 server_base.cc:1034] running on GCE node
W20260620 12:26:38.020012 11637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:26:38.020906 11632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:26:38.021987 11632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:26:38.023150 11632 hybrid_clock.cc:648] HybridClock initialized: now 1781958398023126 us; error 39 us; skew 500 ppm
Jun 20 12:26:38 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:26:38.025830 11632 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:26:38.026744 11632 webserver.cc:460] Webserver started at http://127.0.98.62:42201/ using document root <none> and password file <none>
I20260620 12:26:38.027276 11632 fs_manager.cc:359] Metadata directory not provided
I20260620 12:26:38.027336 11632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:26:38.027544 11632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:26:38 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:26:38.083666 11632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "bbdef73f0ee44d2dbcea6166dc10ee99"
format_stamp: "Formatted at 2026-06-20 12:26:38 on dist-test-slave-bwpp"
server_key: "1b6d62ade3237f24606fc6aff7dca664"
server_key_iv: "e66ee51add6a9c70ddee697467e4d331"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:38.084441 11632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbdef73f0ee44d2dbcea6166dc10ee99"
format_stamp: "Formatted at 2026-06-20 12:26:38 on dist-test-slave-bwpp"
server_key: "1b6d62ade3237f24606fc6aff7dca664"
server_key_iv: "e66ee51add6a9c70ddee697467e4d331"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:38.089380 11632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:26:38.103180 11647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:38.104631 11632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:26:38.104776 11632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "bbdef73f0ee44d2dbcea6166dc10ee99"
format_stamp: "Formatted at 2026-06-20 12:26:38 on dist-test-slave-bwpp"
server_key: "1b6d62ade3237f24606fc6aff7dca664"
server_key_iv: "e66ee51add6a9c70ddee697467e4d331"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:38.104915 11632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:26:38.126616 11632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:26:38.129707 11632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:26:38.129906 11632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:26:38.138146 11632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45235
I20260620 12:26:38.138170 11699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45235 every 8 connection(s)
I20260620 12:26:38.139218 11632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:26:38.141979 11700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:26:38.145133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 11632
I20260620 12:26:38.145282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:26:38.145619 392 external_mini_cluster.cc:1428] Setting key 31474887c909550e4a45ec85ddf68c4e
I20260620 12:26:38.149924 11700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99: Bootstrap starting.
I20260620 12:26:38.152726 11700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99: Neither blocks nor log segments found. Creating new log.
I20260620 12:26:38.153990 11700 log.cc:824] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99: Log is configured to *not* fsync() on all Append() calls
I20260620 12:26:38.156255 11700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99: No bootstrap required, opened a new log
Jun 20 12:26:38 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:26:38.160945 11700 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45235 } }
I20260620 12:26:38.161216 11700 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:26:38.161273 11700 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdef73f0ee44d2dbcea6166dc10ee99, State: Initialized, Role: FOLLOWER
I20260620 12:26:38.161808 11700 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45235 } }
I20260620 12:26:38.161978 11700 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:26:38.162045 11700 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:26:38.162154 11700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:26:38.163110 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:38.147753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.0.98.62:45235)
0620 12:26:38.148684 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:38.148705 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:38.148747 (+ 42us) server_negotiation.cc:407] Connection header received
0620 12:26:38.149557 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:38.149581 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:38.150026 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:38.150383 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:26:38.151859 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:38.153688 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:38.154614 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:38.155175 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:38.155299 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:38.158097 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:26:38.158103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:26:38.158148 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:26:38.158180 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:26:38.160005 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:38.160436 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:38.160442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:38.160448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:38.160586 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:38.160837 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:38.160840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:38.160841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:38.161350 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:26:38.161546 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:38.161833 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 12:26:38.162121 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":113,"threads_started":1}
I20260620 12:26:38.163322 11700 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45235 } }
I20260620 12:26:38.163679 11700 leader_election.cc:302] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbdef73f0ee44d2dbcea6166dc10ee99; no voters:
I20260620 12:26:38.164012 11700 leader_election.cc:288] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:26:38.164124 11705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:26:38.164371 11705 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [term 1 LEADER]: Becoming Leader. State: Replica: bbdef73f0ee44d2dbcea6166dc10ee99, State: Running, Role: LEADER
I20260620 12:26:38.164678 11705 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45235 } }
I20260620 12:26:38.165259 11700 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:26:38.166463 11706 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45235 } } }
I20260620 12:26:38.166509 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbdef73f0ee44d2dbcea6166dc10ee99. Latest consensus state: current_term: 1 leader_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdef73f0ee44d2dbcea6166dc10ee99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45235 } } }
I20260620 12:26:38.166584 11706 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [sys.catalog]: This master's current role is: LEADER
I20260620 12:26:38.166610 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99 [sys.catalog]: This master's current role is: LEADER
I20260620 12:26:38.166958 11714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:26:38.169880 11714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:26:38.170298 11700 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:26:38.179083 11714 catalog_manager.cc:1347] Generated new cluster ID: d361cc7e8b594f89ae2d3d00297d305c
I20260620 12:26:38.179162 11714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:26:38.189535 11714 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:26:38.190806 11714 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:26:38.197281 11714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99: Generated new TSK 0
I20260620 12:26:38.198616 11714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:26:38.838327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45235
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:26:38.957406 11745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:26:38.957693 11745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:26:38.957773 11745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:26:38.961869 11745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:26:38.961951 11745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:26:38.962054 11745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:26:38.967905 11745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:50093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:26:38.969056 11745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:26:38.970230 11745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:26:38.980491 11753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:26:38.980507 11751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:26:38.980749 11745 server_base.cc:1034] running on GCE node
W20260620 12:26:38.981078 11750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:26:38.981480 11745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:26:38.982196 11745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:26:38.983412 11745 hybrid_clock.cc:648] HybridClock initialized: now 1781958398983390 us; error 52 us; skew 500 ppm
Jun 20 12:26:38 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:26:38.986821 11745 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:26:38.987802 11745 webserver.cc:460] Webserver started at http://127.0.98.1:45073/ using document root <none> and password file <none>
I20260620 12:26:38.988387 11745 fs_manager.cc:359] Metadata directory not provided
I20260620 12:26:38.988478 11745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:26:38.988780 11745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:26:38 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:26:39.005277 11745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "edbdb2f4aa2344358aa74e6c00492623"
format_stamp: "Formatted at 2026-06-20 12:26:38 on dist-test-slave-bwpp"
server_key: "678d1ae303cfe39183a4fc19c6af3c8e"
server_key_iv: "5b61f00e06de2606bfefa2eedc04bbb0"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.006007 11745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "edbdb2f4aa2344358aa74e6c00492623"
format_stamp: "Formatted at 2026-06-20 12:26:38 on dist-test-slave-bwpp"
server_key: "678d1ae303cfe39183a4fc19c6af3c8e"
server_key_iv: "5b61f00e06de2606bfefa2eedc04bbb0"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.011030 11745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 12:26:39.028971 11762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.030829 11745 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 12:26:39.030969 11745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "edbdb2f4aa2344358aa74e6c00492623"
format_stamp: "Formatted at 2026-06-20 12:26:38 on dist-test-slave-bwpp"
server_key: "678d1ae303cfe39183a4fc19c6af3c8e"
server_key_iv: "5b61f00e06de2606bfefa2eedc04bbb0"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.031124 11745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:26:39.054123 11745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:26:39.057894 11745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:26:39.058084 11745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:26:39.058812 11745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:26:39.059846 11745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:26:39.059911 11745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.059970 11745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:26:39.059998 11745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.070462 11745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43527
I20260620 12:26:39.071602 11745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:26:39.075740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 11745
I20260620 12:26:39.075911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:26:39.076252 392 external_mini_cluster.cc:1428] Setting key 4da730c929e5c9bba98ed633ec8516a4
I20260620 12:26:39.080962 11875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43527 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:26:39.115607 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.086356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35797 (local address 127.0.98.62:45235)
0620 12:26:39.086729 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:39.086740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:39.092137 (+ 5397us) server_negotiation.cc:407] Connection header received
0620 12:26:39.093353 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:39.093357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:39.095514 (+ 2157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:39.095612 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:26:39.097247 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.098116 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.106443 (+ 8327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.106940 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.107029 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.110695 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:26:39.110700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:26:39.110703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:26:39.110734 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:26:39.113047 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.113690 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.113694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.113696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.113780 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.114203 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.114207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.114209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.114444 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:26:39.114523 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:39.115258 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 12:26:39.115404 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":120,"threads_started":1}
I20260620 12:26:39.117494 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.085208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45235 (local address 127.0.98.1:35797)
0620 12:26:39.091471 (+ 6263us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:39.091539 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:39.092968 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:39.095954 (+ 2986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:39.095965 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:39.096339 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:26:39.097058 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.097077 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.098436 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.098440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.099878 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.099888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.107283 (+ 7395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.107287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.107534 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.108400 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:26:39.108424 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:26:39.110453 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:26:39.113192 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:26:39.113198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:26:39.113217 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:26:39.113573 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:26:39.113871 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:26:39.113876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:26:39.113880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:26:39.114073 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:26:39.114586 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:26:39.114593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:26:39.114968 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 12:26:39.116078 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0620 12:26:39.116442 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5960,"thread_start_us":139,"threads_started":1}
I20260620 12:26:39.119576 11876 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45235
I20260620 12:26:39.119828 11876 heartbeater.cc:464] Registering TS with master...
I20260620 12:26:39.120440 11876 heartbeater.cc:511] Master 127.0.98.62:45235 requested a full tablet report, sending...
I20260620 12:26:39.122172 11664 ts_manager.cc:194] Registered new tserver with Master: edbdb2f4aa2344358aa74e6c00492623 (127.0.98.1:43527)
I20260620 12:26:39.123744 11664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:26:39.133472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45235
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 12:26:39.291559 11884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:26:39.291916 11884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:26:39.291982 11884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:26:39.297936 11884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:26:39.298029 11884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:26:39.298130 11884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:26:39.306079 11884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:50093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:26:39.307514 11884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:26:39.308862 11884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:26:39.320732 11890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:26:39.320824 11893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:26:39.320744 11891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:26:39.321094 11884 server_base.cc:1034] running on GCE node
I20260620 12:26:39.321539 11884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:26:39.322330 11884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:26:39.323530 11884 hybrid_clock.cc:648] HybridClock initialized: now 1781958399323504 us; error 42 us; skew 500 ppm
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:26:39.327037 11884 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:26:39.328099 11884 webserver.cc:460] Webserver started at http://127.0.98.2:38383/ using document root <none> and password file <none>
I20260620 12:26:39.328794 11884 fs_manager.cc:359] Metadata directory not provided
I20260620 12:26:39.328882 11884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:26:39.329139 11884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:26:39.352844 11884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "0d13b7d3f43f47a0a3e48c4542c4baea"
format_stamp: "Formatted at 2026-06-20 12:26:39 on dist-test-slave-bwpp"
server_key: "d253ee1da51315426b16034db2162f5d"
server_key_iv: "c1ec8e0b1ea8cb5737300a7752809dc1"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.354143 11884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0d13b7d3f43f47a0a3e48c4542c4baea"
format_stamp: "Formatted at 2026-06-20 12:26:39 on dist-test-slave-bwpp"
server_key: "d253ee1da51315426b16034db2162f5d"
server_key_iv: "c1ec8e0b1ea8cb5737300a7752809dc1"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.360061 11884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:26:39.378512 11900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.380175 11884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 12:26:39.380329 11884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "0d13b7d3f43f47a0a3e48c4542c4baea"
format_stamp: "Formatted at 2026-06-20 12:26:39 on dist-test-slave-bwpp"
server_key: "d253ee1da51315426b16034db2162f5d"
server_key_iv: "c1ec8e0b1ea8cb5737300a7752809dc1"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.380491 11884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:26:39.410225 11884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:26:39.424067 11884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:26:39.424290 11884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:26:39.424937 11884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:26:39.425892 11884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:26:39.425958 11884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.426013 11884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:26:39.426043 11884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.437160 11884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42489
I20260620 12:26:39.438867 11884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:26:39.440268 12013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42489 every 8 connection(s)
I20260620 12:26:39.444490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 11884
I20260620 12:26:39.444615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:26:39.444929 392 external_mini_cluster.cc:1428] Setting key f879c4378f393f68413c2967983c0577
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:26:39.461480 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.442180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44451 (local address 127.0.98.62:45235)
0620 12:26:39.442319 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:39.442323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:39.445741 (+ 3418us) server_negotiation.cc:407] Connection header received
0620 12:26:39.449586 (+ 3845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:39.449590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:39.449644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:39.449695 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:26:39.451404 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.452257 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.454011 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.454337 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.454463 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.457438 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:26:39.457442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:26:39.457445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:26:39.457472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:26:39.459189 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.459695 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.459699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.459701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.459789 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.460151 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.460154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.460156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.460370 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:26:39.460488 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:39.461127 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 12:26:39.461295 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:26:39.462540 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.442723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45235 (local address 127.0.98.2:44451)
0620 12:26:39.445577 (+ 2854us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:39.445635 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:39.449305 (+ 3670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:39.450121 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:39.450131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:39.450511 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:26:39.451213 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.451232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.452436 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.452440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.453854 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.453865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.454670 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.454675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.454886 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.455585 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:26:39.455605 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:26:39.457233 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:26:39.459305 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:26:39.459320 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:26:39.459333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:26:39.459581 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:26:39.459890 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:26:39.459893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:26:39.459895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:26:39.460048 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:26:39.460481 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:26:39.460486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:26:39.460867 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 12:26:39.461027 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 12:26:39.461484 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2622,"thread_start_us":148,"threads_started":1}
I20260620 12:26:39.463796 12014 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45235
I20260620 12:26:39.464035 12014 heartbeater.cc:464] Registering TS with master...
I20260620 12:26:39.464542 12014 heartbeater.cc:511] Master 127.0.98.62:45235 requested a full tablet report, sending...
I20260620 12:26:39.465570 11664 ts_manager.cc:194] Registered new tserver with Master: 0d13b7d3f43f47a0a3e48c4542c4baea (127.0.98.2:42489)
I20260620 12:26:39.466182 11664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:26:39.481725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45235
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:26:39.622095 12019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:26:39.622570 12019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:26:39.622735 12019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:26:39.629284 12019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:26:39.629467 12019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:26:39.629856 12019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:26:39.638371 12019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:50093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:26:39.640038 12019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:26:39.641417 12019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 12:26:39.659373 12027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:26:39.659737 12025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:26:39.660053 12019 server_base.cc:1034] running on GCE node
W20260620 12:26:39.660733 12024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:26:39.661335 12019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:26:39.662109 12019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:26:39.665390 12019 hybrid_clock.cc:648] HybridClock initialized: now 1781958399665312 us; error 86 us; skew 500 ppm
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:26:39.668948 12019 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:26:39.670115 12019 webserver.cc:460] Webserver started at http://127.0.98.3:43817/ using document root <none> and password file <none>
I20260620 12:26:39.670882 12019 fs_manager.cc:359] Metadata directory not provided
I20260620 12:26:39.670975 12019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:26:39.671222 12019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:26:39.686587 12019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "94e8fc45206f427fb9142128fe439893"
format_stamp: "Formatted at 2026-06-20 12:26:39 on dist-test-slave-bwpp"
server_key: "08e074978ce94f70831690014eff99cb"
server_key_iv: "f6d00ec29b6e5e9774bf639f7127eed2"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.687291 12019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94e8fc45206f427fb9142128fe439893"
format_stamp: "Formatted at 2026-06-20 12:26:39 on dist-test-slave-bwpp"
server_key: "08e074978ce94f70831690014eff99cb"
server_key_iv: "f6d00ec29b6e5e9774bf639f7127eed2"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.709355 12019 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.002s sys 0.002s
I20260620 12:26:39.735128 12034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.737118 12019 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.004s
I20260620 12:26:39.737257 12019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "94e8fc45206f427fb9142128fe439893"
format_stamp: "Formatted at 2026-06-20 12:26:39 on dist-test-slave-bwpp"
server_key: "08e074978ce94f70831690014eff99cb"
server_key_iv: "f6d00ec29b6e5e9774bf639f7127eed2"
server_key_version: "kuduclusterkey@0"
I20260620 12:26:39.737404 12019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:26:39.753973 12019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:26:39.757575 12019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:26:39.757783 12019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:26:39.758498 12019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:26:39.759546 12019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:26:39.759610 12019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.760010 12019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:26:39.760054 12019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:26:39.771064 12019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44733
I20260620 12:26:39.772271 12019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:26:39.776818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 12019
I20260620 12:26:39.776954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:26:39.777242 392 external_mini_cluster.cc:1428] Setting key 22ca5ebda6c3655aa93cba2b64d5b3e1
I20260620 12:26:39.783118 12147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44733 every 8 connection(s)
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:26:39.797135 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.784990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52781 (local address 127.0.98.62:45235)
0620 12:26:39.785138 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:39.785143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:39.786162 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 12:26:39.787225 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:39.787228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:39.787280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:39.787363 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:26:39.788465 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.789034 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.790469 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.790812 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.790915 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.793536 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:26:39.793540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:26:39.793543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:26:39.793571 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:26:39.795118 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.795610 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.795614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.795616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.795705 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.796032 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.796036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.796037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.796255 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:26:39.796357 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:39.796853 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 12:26:39.796976 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:26:39.797906 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.785507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45235 (local address 127.0.98.3:52781)
0620 12:26:39.785995 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:39.786049 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:39.787021 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:39.787554 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:39.787561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:39.787832 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:26:39.788326 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.788339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.789163 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.789171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.790327 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.790336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.791075 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.791078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.791268 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.791875 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:26:39.791898 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:26:39.793374 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:26:39.795233 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:26:39.795238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:26:39.795250 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:26:39.795510 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:26:39.795796 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:26:39.795799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:26:39.795800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:26:39.795938 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:26:39.796368 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:26:39.796373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:26:39.796615 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 12:26:39.796770 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 12:26:39.797018 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":150,"threads_started":1}
I20260620 12:26:39.799105 12148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45235
I20260620 12:26:39.799360 12148 heartbeater.cc:464] Registering TS with master...
I20260620 12:26:39.799860 12148 heartbeater.cc:511] Master 127.0.98.62:45235 requested a full tablet report, sending...
I20260620 12:26:39.800935 11664 ts_manager.cc:194] Registered new tserver with Master: 94e8fc45206f427fb9142128fe439893 (127.0.98.3:44733)
I20260620 12:26:39.801489 11664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52781
I20260620 12:26:39.804395 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:26:39.815853 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.806497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.0.98.62:45235)
0620 12:26:39.806635 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:39.806640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:39.806864 (+ 224us) server_negotiation.cc:407] Connection header received
0620 12:26:39.806960 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:39.806963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:39.807008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:39.807075 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:26:39.807781 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.808596 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.809899 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.810257 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.810379 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.812141 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:26:39.812146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:26:39.812149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:26:39.812176 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:26:39.813813 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.814205 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.814211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.814212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.814308 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:26:39.814557 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:26:39.814561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:26:39.814563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:26:39.815208 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:26:39.815355 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:39.815549 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 12:26:39.815671 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:26:39.819113 11664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:26:39.823496 11664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:26:39.843091 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.834119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42489 (local address 127.0.0.1:46230)
0620 12:26:39.836204 (+ 2085us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:39.836215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:39.836286 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:39.836915 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:39.836918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:39.836947 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:26:39.837043 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.837707 (+ 664us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.839179 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.839182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.841235 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.841245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.842512 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.842515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.842628 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.842688 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 12:26:39.842809 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 12:26:39.842921 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2033,"thread_start_us":81,"threads_started":1}
I20260620 12:26:39.844694 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.834704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.98.2:42489)
0620 12:26:39.835441 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:39.835452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:39.836376 (+ 924us) server_negotiation.cc:407] Connection header received
0620 12:26:39.836455 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:39.836460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:39.836645 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:39.836785 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:26:39.837827 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.839049 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.841407 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.842218 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.842315 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.842453 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:39.843705 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0620 12:26:39.843809 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":87,"threads_started":1}
I20260620 12:26:39.846823 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.833710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43527 (local address 127.0.0.1:40030)
0620 12:26:39.835942 (+ 2232us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:39.835954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:39.836037 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:39.838722 (+ 2685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:39.838728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:39.838744 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:26:39.838826 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.838831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.840732 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.840735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.841986 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.841998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.846418 (+ 4420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.846422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.846527 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.846548 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:26:39.846616 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:26:39.846697 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2166,"thread_start_us":72,"threads_started":1}
I20260620 12:26:39.847395 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.834349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.0.98.1:43527)
0620 12:26:39.835248 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:39.835256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:39.838170 (+ 2914us) server_negotiation.cc:407] Connection header received
0620 12:26:39.838243 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:39.838248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:39.838435 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:39.838559 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:26:39.838960 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.840595 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.845101 (+ 4506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.846109 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.846211 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.846342 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:39.847146 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 12:26:39.847263 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":90,"threads_started":1}
I20260620 12:26:39.848167 11938 tablet_service.cc:1467] Processing CreateTablet for tablet 146b1d2ea31c4421a4d5c4dccdb0f816 (DEFAULT_TABLE table=test-table [id=e0fd36119dfd42d0be384d4ecaf37911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:26:39.849352 11938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146b1d2ea31c4421a4d5c4dccdb0f816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:26:39.849571 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.833367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44733 (local address 127.0.0.1:42204)
0620 12:26:39.835524 (+ 2157us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:39.835572 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:39.835708 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:39.836453 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:39.836460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:39.836485 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:26:39.836623 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.836630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.842138 (+ 5508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.842141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.843588 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:39.843599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.844741 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.844746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:39.844861 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.844887 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:26:39.844936 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 12:26:39.849360 (+ 4424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1984,"thread_start_us":103,"threads_started":1}
I20260620 12:26:39.850442 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:39.833487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.0.98.3:44733)
0620 12:26:39.835250 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:39.835261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:39.835720 (+ 459us) server_negotiation.cc:407] Connection header received
0620 12:26:39.835852 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:39.835858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:39.836084 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:39.836256 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:26:39.836842 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.838057 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.843769 (+ 5712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:39.844598 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:39.850012 (+ 5414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:39.850158 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:39.850208 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 12:26:39.850298 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1658,"thread_start_us":105,"threads_started":1}
I20260620 12:26:39.852586 11810 tablet_service.cc:1467] Processing CreateTablet for tablet 146b1d2ea31c4421a4d5c4dccdb0f816 (DEFAULT_TABLE table=test-table [id=e0fd36119dfd42d0be384d4ecaf37911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:26:39.852994 12074 tablet_service.cc:1467] Processing CreateTablet for tablet 146b1d2ea31c4421a4d5c4dccdb0f816 (DEFAULT_TABLE table=test-table [id=e0fd36119dfd42d0be384d4ecaf37911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:26:39.853672 11810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146b1d2ea31c4421a4d5c4dccdb0f816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:26:39.853852 12074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146b1d2ea31c4421a4d5c4dccdb0f816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:26:39.861760 12168 tablet_bootstrap.cc:492] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623: Bootstrap starting.
I20260620 12:26:39.861761 12167 tablet_bootstrap.cc:492] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Bootstrap starting.
I20260620 12:26:39.864097 12168 tablet_bootstrap.cc:654] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623: Neither blocks nor log segments found. Creating new log.
I20260620 12:26:39.864549 12167 tablet_bootstrap.cc:654] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Neither blocks nor log segments found. Creating new log.
I20260620 12:26:39.865036 12168 log.cc:824] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623: Log is configured to *not* fsync() on all Append() calls
I20260620 12:26:39.865352 12167 log.cc:824] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Log is configured to *not* fsync() on all Append() calls
I20260620 12:26:39.867125 12168 tablet_bootstrap.cc:492] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623: No bootstrap required, opened a new log
I20260620 12:26:39.867125 12167 tablet_bootstrap.cc:492] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: No bootstrap required, opened a new log
I20260620 12:26:39.867312 12167 ts_tablet_manager.cc:1397] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:26:39.867326 12168 ts_tablet_manager.cc:1397] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:26:39.871407 12167 raft_consensus.cc:348] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:39.871407 12168 raft_consensus.cc:348] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:39.871685 12167 raft_consensus.cc:374] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:26:39.871685 12168 raft_consensus.cc:374] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:26:39.871735 12167 raft_consensus.cc:729] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e8fc45206f427fb9142128fe439893, State: Initialized, Role: FOLLOWER
I20260620 12:26:39.871735 12168 raft_consensus.cc:729] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbdb2f4aa2344358aa74e6c00492623, State: Initialized, Role: FOLLOWER
I20260620 12:26:39.872257 12167 consensus_queue.cc:260] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:39.872529 12168 consensus_queue.cc:260] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:39.873343 12167 ts_tablet_manager.cc:1428] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:26:39.873466 12168 ts_tablet_manager.cc:1428] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:26:39.873734 12148 heartbeater.cc:503] Master 127.0.98.62:45235 was elected leader, sending a full tablet report...
I20260620 12:26:39.873780 11876 heartbeater.cc:503] Master 127.0.98.62:45235 was elected leader, sending a full tablet report...
I20260620 12:26:39.874365 12166 tablet_bootstrap.cc:492] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Bootstrap starting.
I20260620 12:26:39.878104 12166 tablet_bootstrap.cc:654] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Neither blocks nor log segments found. Creating new log.
I20260620 12:26:39.879491 12166 log.cc:824] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Log is configured to *not* fsync() on all Append() calls
I20260620 12:26:39.881820 12166 tablet_bootstrap.cc:492] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: No bootstrap required, opened a new log
I20260620 12:26:39.882047 12166 ts_tablet_manager.cc:1397] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260620 12:26:39.885036 12166 raft_consensus.cc:348] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:39.885288 12166 raft_consensus.cc:374] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:26:39.885347 12166 raft_consensus.cc:729] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d13b7d3f43f47a0a3e48c4542c4baea, State: Initialized, Role: FOLLOWER
I20260620 12:26:39.885829 12166 consensus_queue.cc:260] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:39.886524 12014 heartbeater.cc:503] Master 127.0.98.62:45235 was elected leader, sending a full tablet report...
I20260620 12:26:39.887888 12166 ts_tablet_manager.cc:1428] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260620 12:26:39.942320 12015 tablet.cc:2369] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:26:39 dist-test-slave-bwpp krb5kdc[11188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 12:26:40.033913 12149 tablet.cc:2369] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:26:40.066195 12171 raft_consensus.cc:482] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:26:40.066473 12171 raft_consensus.cc:504] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:40.067907 12171 leader_election.cc:288] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d13b7d3f43f47a0a3e48c4542c4baea (127.0.98.2:42489), edbdb2f4aa2344358aa74e6c00492623 (127.0.98.1:43527)
I20260620 12:26:40.073558 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.068344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42489 (local address 127.0.98.3:33857)
0620 12:26:40.068968 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:40.068983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:40.069068 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:40.069775 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:40.069778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:40.069807 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:26:40.069924 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.069931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.071031 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.071035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.072053 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.072062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.073104 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.073108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.073211 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.073234 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:26:40.073339 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 12:26:40.073413 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":4}
I20260620 12:26:40.073889 12174 raft_consensus.cc:482] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:26:40.074070 12174 raft_consensus.cc:504] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:40.075371 12174 leader_election.cc:288] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94e8fc45206f427fb9142128fe439893 (127.0.98.3:44733), edbdb2f4aa2344358aa74e6c00492623 (127.0.98.1:43527)
I20260620 12:26:40.077136 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.069320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33857 (local address 127.0.98.2:42489)
0620 12:26:40.069443 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:40.069448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:40.069463 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:26:40.069518 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:40.069521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:40.069570 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:40.069647 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:26:40.070085 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.070888 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.072236 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.072864 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.072974 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.073044 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:40.076942 (+ 3898us) server_negotiation.cc:299] Negotiation successful
0620 12:26:40.077023 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:26:40.077657 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b1d2ea31c4421a4d5c4dccdb0f816" candidate_uuid: "94e8fc45206f427fb9142128fe439893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" is_pre_election: true
I20260620 12:26:40.077870 11968 raft_consensus.cc:2454] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e8fc45206f427fb9142128fe439893 in term 0.
I20260620 12:26:40.078344 12036 leader_election.cc:302] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d13b7d3f43f47a0a3e48c4542c4baea, 94e8fc45206f427fb9142128fe439893; no voters:
I20260620 12:26:40.078591 12171 raft_consensus.cc:2781] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:26:40.078696 12171 raft_consensus.cc:482] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:26:40.078747 12171 raft_consensus.cc:3037] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:26:40.079515 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.069478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33187 (local address 127.0.98.1:43527)
0620 12:26:40.069595 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:40.069601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:40.070839 (+ 1238us) server_negotiation.cc:407] Connection header received
0620 12:26:40.071037 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:40.071040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:40.071089 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:40.071136 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:26:40.071543 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.072358 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.077920 (+ 5562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.078571 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.078724 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.078788 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:40.079312 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 12:26:40.079393 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:26:40.079949 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.068603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43527 (local address 127.0.98.3:33187)
0620 12:26:40.070747 (+ 2144us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:40.070762 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:40.070904 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:40.071268 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:40.071271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:40.071287 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:26:40.071402 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.071408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.074435 (+ 3027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.074438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.076484 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.076496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.079062 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.079066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.079168 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.079189 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:26:40.079717 (+ 528us) client_negotiation.cc:240] Negotiation successful
0620 12:26:40.079791 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2039,"thread_start_us":111,"threads_started":1}
I20260620 12:26:40.080134 12171 raft_consensus.cc:504] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:40.080554 12171 leader_election.cc:288] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [CANDIDATE]: Term 1 election: Requested vote from peers 0d13b7d3f43f47a0a3e48c4542c4baea (127.0.98.2:42489), edbdb2f4aa2344358aa74e6c00492623 (127.0.98.1:43527)
I20260620 12:26:40.080756 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b1d2ea31c4421a4d5c4dccdb0f816" candidate_uuid: "94e8fc45206f427fb9142128fe439893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbdb2f4aa2344358aa74e6c00492623" is_pre_election: true
I20260620 12:26:40.080972 11817 raft_consensus.cc:2454] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e8fc45206f427fb9142128fe439893 in term 0.
I20260620 12:26:40.081516 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b1d2ea31c4421a4d5c4dccdb0f816" candidate_uuid: "94e8fc45206f427fb9142128fe439893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbdb2f4aa2344358aa74e6c00492623"
I20260620 12:26:40.081621 11817 raft_consensus.cc:3037] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:26:40.081897 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b1d2ea31c4421a4d5c4dccdb0f816" candidate_uuid: "94e8fc45206f427fb9142128fe439893" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea"
I20260620 12:26:40.081998 11968 raft_consensus.cc:3037] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:26:40.082953 11817 raft_consensus.cc:2454] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e8fc45206f427fb9142128fe439893 in term 1.
I20260620 12:26:40.083249 11968 raft_consensus.cc:2454] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e8fc45206f427fb9142128fe439893 in term 1.
I20260620 12:26:40.083340 12038 leader_election.cc:302] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94e8fc45206f427fb9142128fe439893, edbdb2f4aa2344358aa74e6c00492623; no voters:
I20260620 12:26:40.083582 12171 raft_consensus.cc:2781] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:26:40.083818 12171 raft_consensus.cc:686] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 1 LEADER]: Becoming Leader. State: Replica: 94e8fc45206f427fb9142128fe439893, State: Running, Role: LEADER
W20260620 12:26:40.084002 11877 tablet.cc:2369] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:26:40.084430 12171 consensus_queue.cc:237] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } }
I20260620 12:26:40.087852 11664 catalog_manager.cc:5495] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94e8fc45206f427fb9142128fe439893 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "94e8fc45206f427fb9142128fe439893" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e8fc45206f427fb9142128fe439893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } health_report { overall_health: UNKNOWN } } }
I20260620 12:26:40.088992 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.076001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42649 (local address 127.0.98.3:44733)
0620 12:26:40.076099 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:40.076103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:40.084642 (+ 8539us) server_negotiation.cc:407] Connection header received
0620 12:26:40.084827 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:40.084830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:40.084875 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:40.084948 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:26:40.085450 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.086157 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.087703 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.088254 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.088340 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.088403 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:40.088820 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 12:26:40.088893 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:26:40.089504 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.075672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44733 (local address 127.0.98.2:42649)
0620 12:26:40.084548 (+ 8876us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:40.084564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:40.084713 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:40.085061 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:40.085064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:40.085081 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:26:40.085195 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.085202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.086354 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.086359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.087565 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.087576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.088583 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.088586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.088679 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.088701 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:26:40.089260 (+ 559us) client_negotiation.cc:240] Negotiation successful
0620 12:26:40.089350 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8791,"thread_start_us":88,"threads_started":1}
I20260620 12:26:40.090415 12102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b1d2ea31c4421a4d5c4dccdb0f816" candidate_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94e8fc45206f427fb9142128fe439893" is_pre_election: true
I20260620 12:26:40.098456 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.076618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55065 (local address 127.0.98.1:43527)
0620 12:26:40.079686 (+ 3068us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:40.079689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:40.090131 (+ 10442us) server_negotiation.cc:407] Connection header received
0620 12:26:40.090323 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:40.090326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:40.090371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:40.090420 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:26:40.096182 (+ 5762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.096673 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.097573 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.097971 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.098081 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.098135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:40.098292 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 12:26:40.098353 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3009,"thread_start_us":109,"threads_started":1}
I20260620 12:26:40.098457 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.076279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43527 (local address 127.0.98.2:55065)
0620 12:26:40.090063 (+ 13784us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:40.090077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:26:40.090192 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:26:40.090616 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:26:40.090619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:26:40.090635 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:26:40.096052 (+ 5417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.096061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.096784 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.096787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.097457 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:26:40.097463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.098079 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.098082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:26:40.098161 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.098187 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:26:40.098267 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 12:26:40.098337 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13694,"thread_start_us":99,"threads_started":1}
I20260620 12:26:40.099093 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146b1d2ea31c4421a4d5c4dccdb0f816" candidate_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbdb2f4aa2344358aa74e6c00492623" is_pre_election: true
I20260620 12:26:40.099272 11817 raft_consensus.cc:2379] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d13b7d3f43f47a0a3e48c4542c4baea in current term 1: Already voted for candidate 94e8fc45206f427fb9142128fe439893 in this term.
I20260620 12:26:40.099740 11904 leader_election.cc:302] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d13b7d3f43f47a0a3e48c4542c4baea; no voters: 94e8fc45206f427fb9142128fe439893, edbdb2f4aa2344358aa74e6c00492623
I20260620 12:26:40.100199 12174 raft_consensus.cc:2726] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 12:26:40.133183 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.128712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.0.98.3:44733)
0620 12:26:40.128833 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:26:40.128838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:26:40.128907 (+ 69us) server_negotiation.cc:407] Connection header received
0620 12:26:40.129143 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:26:40.129146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:26:40.129222 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:26:40.129353 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:26:40.129729 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.130523 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.131513 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:26:40.131752 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:26:40.131895 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:26:40.132214 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:26:40.132794 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:26:40.132937 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:26:40.132974 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 12:26:40.133030 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:26:40.155689 11817 raft_consensus.cc:1261] T 146b1d2ea31c4421a4d5c4dccdb0f816 P edbdb2f4aa2344358aa74e6c00492623 [term 1 FOLLOWER]: Refusing update from remote peer 94e8fc45206f427fb9142128fe439893: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:26:40.156534 12171 consensus_queue.cc:1035] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edbdb2f4aa2344358aa74e6c00492623" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:26:40.163610 11968 raft_consensus.cc:1261] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 1 FOLLOWER]: Refusing update from remote peer 94e8fc45206f427fb9142128fe439893: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:26:40.164387 12171 consensus_queue.cc:1035] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13b7d3f43f47a0a3e48c4542c4baea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:26:40.181381 12184 mvcc.cc:205] Tried to move back new op lower bound from 7298901607028568064 to 7298901606748676096. Current Snapshot: MvccSnapshot[applied={T|T < 7298901607028568064}]
I20260620 12:26:40.181614 12185 mvcc.cc:205] Tried to move back new op lower bound from 7298901607028568064 to 7298901606748676096. Current Snapshot: MvccSnapshot[applied={T|T < 7298901607028568064}]
I20260620 12:26:40.184618 12187 mvcc.cc:205] Tried to move back new op lower bound from 7298901607028568064 to 7298901606748676096. Current Snapshot: MvccSnapshot[applied={T|T < 7298901607028568064}]
I20260620 12:26:40.195320 11664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:26:40.195502 11664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:26:40.199337 11664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbdef73f0ee44d2dbcea6166dc10ee99: Sending DeleteTablet for 3 replicas of tablet 146b1d2ea31c4421a4d5c4dccdb0f816
I20260620 12:26:40.200178 11810 tablet_service.cc:1514] Processing DeleteTablet for tablet 146b1d2ea31c4421a4d5c4dccdb0f816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:26:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40030
I20260620 12:26:40.200387 12074 tablet_service.cc:1514] Processing DeleteTablet for tablet 146b1d2ea31c4421a4d5c4dccdb0f816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:26:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42204
I20260620 12:26:40.200806 11938 tablet_service.cc:1514] Processing DeleteTablet for tablet 146b1d2ea31c4421a4d5c4dccdb0f816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:26:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46230
I20260620 12:26:40.201851 12195 tablet_replica.cc:330] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: stopping tablet replica
I20260620 12:26:40.202208 12195 raft_consensus.cc:2229] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:26:40.202236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 11745
I20260620 12:26:40.203497 12195 raft_consensus.cc:2258] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:26:40.205138 12195 ts_tablet_manager.cc:1905] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:26:40.208657 12195 ts_tablet_manager.cc:1918] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:26:40.208824 12195 log.cc:1196] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/146b1d2ea31c4421a4d5c4dccdb0f816
I20260620 12:26:40.209306 12195 ts_tablet_manager.cc:1939] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 0d13b7d3f43f47a0a3e48c4542c4baea: Deleting consensus metadata
I20260620 12:26:40.210777 11649 catalog_manager.cc:4841] TS 0d13b7d3f43f47a0a3e48c4542c4baea (127.0.98.2:42489): tablet 146b1d2ea31c4421a4d5c4dccdb0f816 (table test-table [id=e0fd36119dfd42d0be384d4ecaf37911]) successfully deleted
I20260620 12:26:40.217370 12194 tablet_replica.cc:330] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: stopping tablet replica
I20260620 12:26:40.217782 12194 raft_consensus.cc:2229] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:26:40.218292 12194 raft_consensus.cc:2258] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:26:40.220356 12194 ts_tablet_manager.cc:1905] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:26:40.223877 12194 ts_tablet_manager.cc:1918] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:26:40.223970 12194 log.cc:1196] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/146b1d2ea31c4421a4d5c4dccdb0f816
I20260620 12:26:40.224354 12194 ts_tablet_manager.cc:1939] T 146b1d2ea31c4421a4d5c4dccdb0f816 P 94e8fc45206f427fb9142128fe439893: Deleting consensus metadata
W20260620 12:26:40.224828 11651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43527 (error 108)
I20260620 12:26:40.225621 11649 catalog_manager.cc:4841] TS 94e8fc45206f427fb9142128fe439893 (127.0.98.3:44733): tablet 146b1d2ea31c4421a4d5c4dccdb0f816 (table test-table [id=e0fd36119dfd42d0be384d4ecaf37911]) successfully deleted
I20260620 12:26:40.225757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 11884
I20260620 12:26:40.231379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 12019
I20260620 12:26:40.233736 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:26:40.233430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43527 (local address 127.0.0.1:40040)
0620 12:26:40.233568 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:26:40.233647 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 12:26:40.233939 11651 catalog_manager.cc:4584] TS edbdb2f4aa2344358aa74e6c00492623 (127.0.98.1:43527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 146b1d2ea31c4421a4d5c4dccdb0f816: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43527: connect: Connection refused (error 111)
I20260620 12:26:40.237445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 11632
I20260620 12:26:40.243434 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 12:26:40.579228 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 12:26:40.579381 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 12:26:40.916167 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 12:26:40.916 UTC [11218] LOG: received smart shutdown request
2026-06-20 12:26:40.920 UTC [11218] LOG: background worker "logical replication launcher" (PID 11227) exited with exit code 1
2026-06-20 12:26:40.920 UTC [11222] LOG: shutting down
2026-06-20 12:26:41.067 UTC [11218] LOG: database system is shut down
2026-06-20T12:26:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46021 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:26:41 dist-test-slave-bwpp krb5kdc[12205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:26:41 dist-test-slave-bwpp krb5kdc[12205](info): set up 2 sockets
Jun 20 12:26:41 dist-test-slave-bwpp krb5kdc[12205](info): commencing operation
krb5kdc: starting...
W20260620 12:26:42.246896 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:26:42 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:26:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 12:26:42.409052 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiQxjdC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/postgres -l logfile start
2026-06-20 12:26:44.742 UTC [12235] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 12:26:44.742 UTC [12235] LOG: listening on IPv4 address "127.0.98.20", port 50469
2026-06-20 12:26:44.745 UTC [12235] LOG: listening on Unix socket "/tmp/.s.PGSQL.50469"
2026-06-20 12:26:44.767 UTC [12238] LOG: database system was shut down at 2026-06-20 12:26:43 UTC
2026-06-20 12:26:44.770 UTC [12235] LOG: database system is ready to accept connections
I20260620 12:26:45.730468 392 mini_postgres.cc:96] Postgres bound to 50469
2026-06-20 12:26:45.736 UTC [12246] FATAL: database "slave" does not exist
127.0.98.20:50469 - accepting connections
I20260620 12:26:45.737140 392 mini_ranger.cc:161] Starting Ranger...
I20260620 12:26:45.751549 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 12:26:46.191442 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 12:26:46.191514 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-admin
2026-06-20 12:26:46,468 [I] DB FLAVOR :POSTGRES
2026-06-20 12:26:46,468 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:26:46,468 [I] Checking connection..
2026-06-20 12:26:46,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 12:26:46,723 [I] Checking connection passed.
2026-06-20 12:26:46,723 [I] --------- Verifying version history table ---------
2026-06-20 12:26:46,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:26:46,969 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:26:46,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:26:47,204 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:26:47,204 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 12:26:47,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 12:26:47.458 UTC [12310] WARNING: there is no transaction in progress
2026-06-20 12:26:47,464 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 12:26:47,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:26:47,720 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 12:26:47,720 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:26:47,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 12:26:47,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 12:26:48,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bwpp.c.gcp-upstream.internal','N') ;"
2026-06-20 12:26:48,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:26:48,688 [I] Table x_portal_user does not exist in database ranger
2026-06-20 12:26:48,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:26:48,931 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 12:26:48,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:26:49,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:26:49,415 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 12:26:49,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 12:26:50.333 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.349 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.365 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.380 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.396 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.414 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.466 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.473 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.487 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.497 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.507 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.519 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.528 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.535 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.541 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:50.548 UTC [12418] WARNING: there is no transaction in progress
2026-06-20 12:26:51,139 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 12:26:51,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 12:26:51,387 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 12:26:51,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:26:51,627 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 12:26:51,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:26:51,883 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 12:26:51,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:26:52,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:26:52,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bwpp.c.gcp-upstream.internal';"
2026-06-20 12:26:52,627 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 12:26:52,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 12:26:52,857 [I] DB_PATCHES have already been applied
I20260620 12:26:52.862805 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiQxjdC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 12:26:52.862884 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 12:26:52.866184 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59659
Jun 20, 2026 12:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 12:26:54.059755 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.323s user 0.000s sys 0.009s
Jun 20, 2026 12:27:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:27:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:27:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:27:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:27:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 12:27:15.316586 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 12:27:15 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958435, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 12:27:15.395558 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 12:27:15.404713 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 12:27:16.206938 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 12:27:16.207027 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms
I20260620 12:27:16.207047 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50469
2026-06-20 12:27:16,374 [I] DB FLAVOR :POSTGRES
2026-06-20 12:27:16,374 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:27:16,374 [I] Checking connection
2026-06-20 12:27:16,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 12:27:16,612 [I] connection success
2026-06-20 12:27:16,613 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 12:27:16,613 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 12:27:16,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 12:27:16,852 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 12:27:16,852 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:27:16,852 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 12:27:16,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 12:27:17,137 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 12:27:17.601402 392 mini_ranger_kms.cc:310] Created kms service
I20260620 12:27:17.819389 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 12:27:17.865223 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 12:27:17 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958435, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:18.231623 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 12:27:18.231782 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiQxjdC/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiQxjdC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 12:27:18.231807 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 12:27:18.235703 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12598
I20260620 12:27:18.235795 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51631
Jun 20, 2026 12:27:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:27:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:27:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:27:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:27:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:27:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 12:27:20.511957 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.116s user 0.009s sys 0.143s
I20260620 12:27:20.512105 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 12:27:20.512219 392 mini_ranger_kms.cc:401] 127.0.98.20:51631/kms/v1/keys
Jun 20 12:27:20 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958440, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 12:27:20 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958440, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:27:21 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958440, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:27:23 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:27:23.487435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36591
--ranger_config_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:23.594937 12649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:23.595172 12649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:23.595211 12649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:23.599192 12649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:23.599246 12649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:23.599267 12649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:23.599303 12649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:23.599334 12649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:23.604782 12649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36591
--ranger_config_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:23.605773 12649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:23.606801 12649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:23.616199 12657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:23.616251 12655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:23.616425 12654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:23.616708 12649 server_base.cc:1034] running on GCE node
I20260620 12:27:23.617262 12649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:23.618305 12649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:23.619491 12649 hybrid_clock.cc:648] HybridClock initialized: now 1781958443619466 us; error 43 us; skew 500 ppm
Jun 20 12:27:23 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:23.625277 12649 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:23.626257 12649 webserver.cc:460] Webserver started at http://127.0.98.62:34331/ using document root <none> and password file <none>
I20260620 12:27:23.626844 12649 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:23.626900 12649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:23.627118 12649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:27:23 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:23.676229 12649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "5022031f963149df99dd39b7bc1fe868"
format_stamp: "Formatted at 2026-06-20 12:27:23 on dist-test-slave-bwpp"
server_key: "cfff62f9e41d6aebd6a3a1d834106155"
server_key_iv: "f8a893c826157e66dbd6816b59175d8e"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:23.676873 12649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "5022031f963149df99dd39b7bc1fe868"
format_stamp: "Formatted at 2026-06-20 12:27:23 on dist-test-slave-bwpp"
server_key: "cfff62f9e41d6aebd6a3a1d834106155"
server_key_iv: "f8a893c826157e66dbd6816b59175d8e"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:23.680816 12649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:27:23.694747 12664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:23.696177 12649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:27:23.696319 12649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "5022031f963149df99dd39b7bc1fe868"
format_stamp: "Formatted at 2026-06-20 12:27:23 on dist-test-slave-bwpp"
server_key: "cfff62f9e41d6aebd6a3a1d834106155"
server_key_iv: "f8a893c826157e66dbd6816b59175d8e"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:23.696473 12649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:23.729678 12649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:23.732795 12649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:23.733009 12649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:23.741204 12649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36591
I20260620 12:27:23.741212 12716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36591 every 8 connection(s)
I20260620 12:27:23.742141 12649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:23.744197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 12649
I20260620 12:27:23.744338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:27:23.744652 392 external_mini_cluster.cc:1428] Setting key e5d548d3ce3740c1fc898bf21e3a4b7f
I20260620 12:27:23.745265 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:23.752072 12717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Bootstrap starting.
I20260620 12:27:23.754806 12717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Neither blocks nor log segments found. Creating new log.
Jun 20 12:27:23 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:23.755584 12717 log.cc:824] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:23.757553 12717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: No bootstrap required, opened a new log
I20260620 12:27:23.761071 12717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:23.761327 12717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:23.761399 12717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5022031f963149df99dd39b7bc1fe868, State: Initialized, Role: FOLLOWER
I20260620 12:27:23.761730 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:23.746048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.0.98.62:36591)
0620 12:27:23.746511 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:23.746530 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:23.746573 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:27:23.747448 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:23.747472 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:23.747868 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:23.748225 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:23.749253 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:23.750875 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:23.752058 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:23.752766 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:23.752917 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:23.755943 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:23.755952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:23.755972 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:23.756003 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:23.758597 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:23.759119 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:23.759124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:23.759130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:23.759257 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:23.759568 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:23.759571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:23.759572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:23.760029 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:23.760201 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:23.760529 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 12:27:23.760793 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260620 12:27:23.761938 12717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:23.762104 12717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:27:23.762169 12717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:27:23.762336 12717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:23.763363 12717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:23.763716 12717 leader_election.cc:302] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5022031f963149df99dd39b7bc1fe868; no voters:
I20260620 12:27:23.764096 12717 leader_election.cc:288] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:27:23.764406 12722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:23.764670 12722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 LEADER]: Becoming Leader. State: Replica: 5022031f963149df99dd39b7bc1fe868, State: Running, Role: LEADER
I20260620 12:27:23.765009 12722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:23.765429 12717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:23.766793 12723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5022031f963149df99dd39b7bc1fe868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } } }
I20260620 12:27:23.766880 12723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:23.766587 12724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5022031f963149df99dd39b7bc1fe868. Latest consensus state: current_term: 1 leader_uuid: "5022031f963149df99dd39b7bc1fe868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } } }
I20260620 12:27:23.767184 12724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:23.767652 12717 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:27:23.768009 12732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:23.773795 12732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:23.788549 12732 catalog_manager.cc:1347] Generated new cluster ID: dd2ed29a84bd442b8406a97109aa2ab2
I20260620 12:27:23.788724 12732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:27:23.812723 12732 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:27:23.814085 12732 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:27:23.829417 12732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Generated new TSK 0
I20260620 12:27:23.830406 12732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:27:24.495949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36591
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:24.639706 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:24.640056 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:24.640118 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:24.644691 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:24.644840 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:24.644970 12762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:27:24.650923 12762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:24.652067 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:24.653092 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:24.662334 12772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:24.662335 12769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:24.662770 12770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:24.663457 12762 server_base.cc:1034] running on GCE node
I20260620 12:27:24.663836 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:24.664465 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:24.667789 12762 hybrid_clock.cc:648] HybridClock initialized: now 1781958444667748 us; error 57 us; skew 500 ppm
Jun 20 12:27:24 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:24.671033 12762 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:27:24.672065 12762 webserver.cc:460] Webserver started at http://127.0.98.1:38235/ using document root <none> and password file <none>
I20260620 12:27:24.672716 12762 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:24.672801 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:24.673009 12762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:27:24 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:24.691735 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd09341189494edf8399b5f788d0dafc"
format_stamp: "Formatted at 2026-06-20 12:27:24 on dist-test-slave-bwpp"
server_key: "6429c2937998f9e88c49db3d3ad563e4"
server_key_iv: "555b50811afa932a76588e6af7c15439"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:24.692737 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd09341189494edf8399b5f788d0dafc"
format_stamp: "Formatted at 2026-06-20 12:27:24 on dist-test-slave-bwpp"
server_key: "6429c2937998f9e88c49db3d3ad563e4"
server_key_iv: "555b50811afa932a76588e6af7c15439"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:24.697801 12762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:27:24.731069 12779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:24.734157 12762 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260620 12:27:24.734300 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "cd09341189494edf8399b5f788d0dafc"
format_stamp: "Formatted at 2026-06-20 12:27:24 on dist-test-slave-bwpp"
server_key: "6429c2937998f9e88c49db3d3ad563e4"
server_key_iv: "555b50811afa932a76588e6af7c15439"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:24.734438 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:24.788236 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:24.792025 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:24.792223 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:24.792865 12762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:24.793861 12762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:24.793920 12762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:24.793972 12762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:24.794055 12762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:24.805290 12762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46629
I20260620 12:27:24.805349 12894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46629 every 8 connection(s)
I20260620 12:27:24.806795 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:27:24.811324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 12762
I20260620 12:27:24.811443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:27:24.811743 392 external_mini_cluster.cc:1428] Setting key 4e03e8b953b2d3c2a663f11710ff49ce
Jun 20 12:27:24 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:27:24.824877 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:24.809101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42877 (local address 127.0.98.62:36591)
0620 12:27:24.809384 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:24.809393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:24.810324 (+ 931us) server_negotiation.cc:407] Connection header received
0620 12:27:24.811621 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:24.811627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:24.811690 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:24.811760 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:24.813846 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:24.814722 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:24.816473 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:24.816800 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:24.816881 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:24.820279 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:24.820283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:24.820286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:24.820314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:24.822244 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:24.822934 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:24.822938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:24.822939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:24.823038 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:24.823473 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:24.823477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:24.823479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:24.823705 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:24.823798 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:24.824550 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 12:27:24.824683 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:27:24.826493 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:24.809571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36591 (local address 127.0.98.1:42877)
0620 12:27:24.810155 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:24.810216 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:24.811354 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:24.812180 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:24.812189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:24.812757 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:24.813672 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:24.813693 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:24.814883 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:24.814887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:24.816326 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:24.816336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:24.817113 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:24.817117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:24.817320 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:24.818055 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:24.818075 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:24.820082 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:24.822375 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:24.822380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:24.822398 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:24.822809 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:24.823160 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:24.823163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:24.823165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:24.823348 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:24.823868 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:24.823875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:24.824224 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 12:27:24.825067 (+ 843us) client_negotiation.cc:240] Negotiation successful
0620 12:27:24.825426 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":141,"threads_started":1}
I20260620 12:27:24.828107 12895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36591
I20260620 12:27:24.828341 12895 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:24.828917 12895 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:24.830931 12681 ts_manager.cc:194] Registered new tserver with Master: cd09341189494edf8399b5f788d0dafc (127.0.98.1:46629)
I20260620 12:27:24.832460 12681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:27:24.847910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36591
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:27:24 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 12:27:24.999751 12901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:25.000164 12901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:25.000283 12901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:25.006289 12901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:25.006363 12901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:25.006445 12901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:27:25.014039 12901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:25.015403 12901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:25.016659 12901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:25.028295 12908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:25.028446 12907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:25.028573 12910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:25.029057 12901 server_base.cc:1034] running on GCE node
I20260620 12:27:25.029516 12901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:25.030294 12901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:25.031503 12901 hybrid_clock.cc:648] HybridClock initialized: now 1781958445031483 us; error 37 us; skew 500 ppm
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:25.035410 12901 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:27:25.036516 12901 webserver.cc:460] Webserver started at http://127.0.98.2:43337/ using document root <none> and password file <none>
I20260620 12:27:25.037281 12901 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:25.037428 12901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:25.037736 12901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:25.053627 12901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "b2ed778160174865b95281502b449d46"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "58552a42aaa8bf4f3f1a90349f938be0"
server_key_iv: "5ce40666f85bb1762a7fd6b5a1fef88a"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:25.054466 12901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b2ed778160174865b95281502b449d46"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "58552a42aaa8bf4f3f1a90349f938be0"
server_key_iv: "5ce40666f85bb1762a7fd6b5a1fef88a"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:25.064023 12901 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260620 12:27:25.090499 12917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:25.092336 12901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:27:25.092494 12901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "b2ed778160174865b95281502b449d46"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "58552a42aaa8bf4f3f1a90349f938be0"
server_key_iv: "5ce40666f85bb1762a7fd6b5a1fef88a"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:25.092659 12901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:25.123162 12901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:25.126874 12901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:25.127121 12901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:25.127853 12901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:25.128916 12901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:25.129017 12901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:25.129117 12901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:25.129180 12901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:25.143015 12901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41873
I20260620 12:27:25.144214 12901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:27:25.150808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 12901
I20260620 12:27:25.150945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:27:25.145684 13030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41873 every 8 connection(s)
I20260620 12:27:25.151283 392 external_mini_cluster.cc:1428] Setting key 727f0068808295651530ba1eb5b9a1ca
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:25.164806 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:25.149380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41655 (local address 127.0.98.62:36591)
0620 12:27:25.149507 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:25.149511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:25.150021 (+ 510us) server_negotiation.cc:407] Connection header received
0620 12:27:25.151665 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:25.151669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:25.151722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:25.151775 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:25.153395 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.154217 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.156000 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.156692 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.156772 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:25.160166 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:25.160171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:25.160174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:25.160204 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:25.162158 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:25.162803 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:25.162807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:25.162809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:25.162934 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:25.163377 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:25.163381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:25.163383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:25.163622 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:25.163720 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:25.164459 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 12:27:25.164596 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:27:25.166745 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:25.148256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36591 (local address 127.0.98.2:41655)
0620 12:27:25.149842 (+ 1586us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:25.149898 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:25.151419 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:25.152174 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:25.152182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:25.152543 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:25.153207 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:25.153227 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.154401 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.154404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:25.155853 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:25.155864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.157092 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.157096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:25.157321 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:25.158007 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:25.158029 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:25.159959 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:25.162302 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:25.162308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:25.162324 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:25.162676 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:25.163061 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:25.163064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:25.163066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:25.163252 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:25.163795 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:25.163801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:25.164152 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 12:27:25.165341 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0620 12:27:25.165680 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1373,"thread_start_us":132,"threads_started":1}
I20260620 12:27:25.168277 13031 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36591
I20260620 12:27:25.168520 13031 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:25.169094 13031 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:25.170317 12681 ts_manager.cc:194] Registered new tserver with Master: b2ed778160174865b95281502b449d46 (127.0.98.2:41873)
I20260620 12:27:25.171160 12681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41655
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:27:25.217378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36591
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 12:27:25.430332 13036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:25.430764 13036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:25.430871 13036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:25.436470 13036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:25.436600 13036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:25.436722 13036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:27:25.444363 13036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:25.445992 13036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:25.447420 13036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:25.461030 13041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:25.461702 13036 server_base.cc:1034] running on GCE node
W20260620 12:27:25.461158 13044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:25.461040 13042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:25.465530 13036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:25.466243 13036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:25.481666 13036 hybrid_clock.cc:648] HybridClock initialized: now 1781958445481582 us; error 110 us; skew 500 ppm
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:25.485901 13036 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:27:25.487128 13036 webserver.cc:460] Webserver started at http://127.0.98.3:34735/ using document root <none> and password file <none>
I20260620 12:27:25.487888 13036 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:25.488032 13036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:25.488307 13036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:25.510910 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "08c5ef436fb1430198805b742bfce777"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "034e8e195f5af33c31030246153017d9"
server_key_iv: "cffa0e481c2b94421d1262afb75722a0"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:25.518297 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "08c5ef436fb1430198805b742bfce777"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "034e8e195f5af33c31030246153017d9"
server_key_iv: "cffa0e481c2b94421d1262afb75722a0"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:25.528118 13036 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.003s
I20260620 12:27:25.558583 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:25.564419 13036 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260620 12:27:25.564563 13036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "08c5ef436fb1430198805b742bfce777"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "034e8e195f5af33c31030246153017d9"
server_key_iv: "cffa0e481c2b94421d1262afb75722a0"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:25.564692 13036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:25.622628 13036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:25.626349 13036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:25.626581 13036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:25.627486 13036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:25.628527 13036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:25.628630 13036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:25.628726 13036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:25.628793 13036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:25.645186 13036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43195
I20260620 12:27:25.646405 13036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:27:25.650261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13036
I20260620 12:27:25.650396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:27:25.650734 392 external_mini_cluster.cc:1428] Setting key 2964a4337570d9161b29286c3f1a3df3
I20260620 12:27:25.660110 13164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43195 every 8 connection(s)
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:25.700775 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:25.662320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41949 (local address 127.0.98.62:36591)
0620 12:27:25.662441 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:25.662445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:25.679160 (+ 16715us) server_negotiation.cc:407] Connection header received
0620 12:27:25.680351 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:25.680354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:25.680412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:25.680501 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:25.683857 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.684677 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.687885 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.688224 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.688341 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:25.693896 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:25.693903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:25.693909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:25.693936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:25.695698 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:25.696452 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:25.696456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:25.696458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:25.696544 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:25.696975 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:25.696979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:25.696981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:25.697204 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:25.697294 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:25.700383 (+ 3089us) server_negotiation.cc:299] Negotiation successful
0620 12:27:25.700509 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:27:25.702822 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:25.661261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36591 (local address 127.0.98.3:41949)
0620 12:27:25.678984 (+ 17723us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:25.679062 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:25.680118 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:25.680887 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:25.680899 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:25.681280 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:25.681995 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:25.682015 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.684846 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.684850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:25.686287 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:25.686320 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:25.688528 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:25.688533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:25.688738 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:25.691874 (+ 3136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:25.691896 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:25.693686 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:25.695951 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:25.695958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:25.695975 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:25.696303 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:25.696650 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:25.696653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:25.696655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:25.696838 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:25.697358 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:25.697365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:25.697687 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 12:27:25.701305 (+ 3618us) client_negotiation.cc:240] Negotiation successful
0620 12:27:25.701645 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17449,"thread_start_us":106,"threads_started":1}
I20260620 12:27:25.704276 13165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36591
I20260620 12:27:25.704516 13165 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:25.705072 13165 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:25.706146 12681 ts_manager.cc:194] Registered new tserver with Master: 08c5ef436fb1430198805b742bfce777 (127.0.98.3:43195)
I20260620 12:27:25.713624 12681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41949
I20260620 12:27:25.714501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:27:25.715406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 12762
I20260620 12:27:25.726656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 12901
I20260620 12:27:25.734782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13036
I20260620 12:27:25.741678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 12649
I20260620 12:27:25.747890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36591
--webserver_interface=127.0.98.62
--webserver_port=34331
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36591
--ranger_config_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:25.887547 13168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:25.887821 13168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:25.887925 13168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:25.891899 13168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:25.891975 13168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:25.892011 13168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:25.892066 13168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:25.892103 13168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:25.897595 13168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36591
--ranger_config_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:25.898617 13168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:25.899647 13168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:25.909108 13173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:25.909093 13174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:25.909093 13176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:25.909909 13168 server_base.cc:1034] running on GCE node
I20260620 12:27:25.910462 13168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:25.911573 13168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:25.912777 13168 hybrid_clock.cc:648] HybridClock initialized: now 1781958445912730 us; error 55 us; skew 500 ppm
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:25.915596 13168 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:25.916421 13168 webserver.cc:460] Webserver started at http://127.0.98.62:34331/ using document root <none> and password file <none>
I20260620 12:27:25.916949 13168 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:25.917042 13168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:27:25 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:25.974852 13168 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:27:25.977321 13184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:25.978775 13168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:27:25.978919 13168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "5022031f963149df99dd39b7bc1fe868"
format_stamp: "Formatted at 2026-06-20 12:27:23 on dist-test-slave-bwpp"
server_key: "cfff62f9e41d6aebd6a3a1d834106155"
server_key_iv: "f8a893c826157e66dbd6816b59175d8e"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:25.979521 13168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:25.998029 13168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:26.000883 13168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:26.001082 13168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:26.009024 13168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36591
I20260620 12:27:26.009090 13236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36591 every 8 connection(s)
I20260620 12:27:26.010134 13168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:26.015475 13237 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:27:26.016311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13168
I20260620 12:27:26.016752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46629
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36591
--webserver_port=38235
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:26.017608 13237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Bootstrap starting.
I20260620 12:27:26.038794 13237 log.cc:824] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:26.046693 13237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:27:26.047115 13237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Bootstrap complete.
I20260620 12:27:26.052556 13237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:26.053037 13237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5022031f963149df99dd39b7bc1fe868, State: Initialized, Role: FOLLOWER
I20260620 12:27:26.053609 13237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:26.053769 13237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:27:26.053864 13237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:27:26.053958 13237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:27:26.056272 13237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:26.056630 13237 leader_election.cc:302] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5022031f963149df99dd39b7bc1fe868; no voters:
I20260620 12:27:26.056943 13237 leader_election.cc:288] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:27:26.057068 13241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:27:26.057413 13241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [term 2 LEADER]: Becoming Leader. State: Replica: 5022031f963149df99dd39b7bc1fe868, State: Running, Role: LEADER
I20260620 12:27:26.057758 13241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260620 12:27:26.058372 13237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:26.058768 13241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5022031f963149df99dd39b7bc1fe868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } } }
I20260620 12:27:26.058987 13241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:26.059290 13241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5022031f963149df99dd39b7bc1fe868. Latest consensus state: current_term: 2 leader_uuid: "5022031f963149df99dd39b7bc1fe868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5022031f963149df99dd39b7bc1fe868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } } }
I20260620 12:27:26.059466 13241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:26.059847 13246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:26.063032 13246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:26.063483 13237 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:27:26.063730 13246 catalog_manager.cc:1259] Loaded cluster ID: dd2ed29a84bd442b8406a97109aa2ab2
I20260620 12:27:26.063831 13246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:27:26.066094 13246 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:27:26.066768 13246 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Loaded TSK: 0
I20260620 12:27:26.067958 13246 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:27:26.168560 13239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:26.168911 13239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:26.168982 13239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:26.176122 13239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:26.176196 13239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:26.176293 13239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:27:26.185043 13239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38235
--tserver_master_addrs=127.0.98.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:26.186260 13239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:26.187712 13239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:26.200702 13277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:26.200716 13275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:26.200850 13274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:26.201552 13239 server_base.cc:1034] running on GCE node
I20260620 12:27:26.203205 13239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:26.204032 13239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:26.205245 13239 hybrid_clock.cc:648] HybridClock initialized: now 1781958446205204 us; error 52 us; skew 500 ppm
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:26.209116 13239 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:27:26.210276 13239 webserver.cc:460] Webserver started at http://127.0.98.1:38235/ using document root <none> and password file <none>
I20260620 12:27:26.211308 13239 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:26.211436 13239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:26.237074 13239 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:27:26.240264 13285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.242103 13239 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 12:27:26.242244 13239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "cd09341189494edf8399b5f788d0dafc"
format_stamp: "Formatted at 2026-06-20 12:27:24 on dist-test-slave-bwpp"
server_key: "6429c2937998f9e88c49db3d3ad563e4"
server_key_iv: "555b50811afa932a76588e6af7c15439"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:26.243072 13239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:26.264796 13239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:26.267920 13239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:26.268131 13239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:26.268759 13239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:26.269753 13239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:26.269805 13239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.269847 13239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:26.269904 13239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.281265 13239 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46629
I20260620 12:27:26.281579 13398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46629 every 8 connection(s)
I20260620 12:27:26.282495 13239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:27:26.291893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13239
I20260620 12:27:26.292382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41873
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36591
--webserver_port=43337
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:26.305339 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:26.285218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36591 (local address 127.0.98.1:35005)
0620 12:27:26.286826 (+ 1608us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:26.286891 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:26.288012 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:26.289813 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:26.289822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:26.290182 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:26.290811 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:26.290827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.292397 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.292403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:26.293888 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:26.293904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.295315 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.295322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:26.295535 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:26.296301 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:26.296326 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:26.298228 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:26.301181 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:26.301190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:26.301207 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:26.301585 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:26.302055 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:26.302060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:26.302065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:26.302256 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:26.303179 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:26.303188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:26.303541 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 12:27:26.303786 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 12:27:26.304177 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"thread_start_us":141,"threads_started":1}
I20260620 12:27:26.305621 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:26.286787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35005 (local address 127.0.98.62:36591)
0620 12:27:26.287253 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:26.287271 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:26.287309 (+ 38us) server_negotiation.cc:407] Connection header received
0620 12:27:26.288518 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:26.288534 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:26.288928 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:26.289279 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:26.290994 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.292227 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.294064 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.294863 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.295043 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:26.298473 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:26.298481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:26.298500 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:26.298541 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:26.301027 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.301740 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.301748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.301756 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.301914 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.302385 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.302391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.302393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.302950 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:26.303113 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:26.303952 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 12:27:26.304284 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
I20260620 12:27:26.308936 13399 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36591
I20260620 12:27:26.309190 13399 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.309803 13399 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
W20260620 12:27:26.312351 13399 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36591 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:27:26.312481 13399 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.312657 13399 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:26.313701 13399 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.313889 13399 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
W20260620 12:27:26.314805 13399 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:27:26.435009 13403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:26.435300 13403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:26.435344 13403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:26.439465 13403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:26.439523 13403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:26.439581 13403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:27:26.445003 13403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43337
--tserver_master_addrs=127.0.98.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:26.446079 13403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:26.447157 13403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:26.456753 13408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:26.456761 13409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:26.457746 13411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:26.457970 13403 server_base.cc:1034] running on GCE node
I20260620 12:27:26.458640 13403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:26.459573 13403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:26.460788 13403 hybrid_clock.cc:648] HybridClock initialized: now 1781958446460760 us; error 45 us; skew 500 ppm
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:26.463593 13403 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:27:26.464599 13403 webserver.cc:460] Webserver started at http://127.0.98.2:43337/ using document root <none> and password file <none>
I20260620 12:27:26.465157 13403 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:26.465260 13403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:26.484547 13403 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:27:26.487169 13419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.488464 13403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:27:26.488612 13403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "b2ed778160174865b95281502b449d46"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "58552a42aaa8bf4f3f1a90349f938be0"
server_key_iv: "5ce40666f85bb1762a7fd6b5a1fef88a"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:26.489156 13403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:26.503249 13403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:26.506213 13403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:26.506407 13403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:26.507086 13403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:26.507989 13403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:26.508059 13403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.508148 13403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:26.508203 13403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.519129 13403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41873
I20260620 12:27:26.519241 13532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41873 every 8 connection(s)
I20260620 12:27:26.520238 13403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:26.530962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13403
I20260620 12:27:26.531494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43195
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36591
--webserver_port=34735
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:26.537578 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:26.522284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46447 (local address 127.0.98.62:36591)
0620 12:27:26.522454 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:26.522461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:26.523315 (+ 854us) server_negotiation.cc:407] Connection header received
0620 12:27:26.524385 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:26.524389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:26.524446 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:26.524503 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:26.526261 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.526886 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.528094 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.528320 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.528436 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:26.533767 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:26.533772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:26.533775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:26.533806 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:26.535518 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.536042 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.536047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.536049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.536131 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.536477 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.536481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.536485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.536667 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:26.536777 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:26.537280 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 12:27:26.537408 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:27:26.538434 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:26.522699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36591 (local address 127.0.98.2:46447)
0620 12:27:26.523168 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:26.523218 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:26.524141 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:26.524921 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:26.524929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:26.525318 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:26.526069 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:26.526093 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.527027 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.527029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:26.527955 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:26.527964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.528472 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.528477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:26.528643 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:26.529333 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:26.529355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:26.533565 (+ 4210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:26.535657 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:26.535663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:26.535678 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:26.535930 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:26.536235 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:26.536240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:26.536242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:26.536380 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:26.536773 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:26.536779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:26.537038 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 12:27:26.537236 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 12:27:26.537551 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260620 12:27:26.539628 13533 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36591
I20260620 12:27:26.539867 13533 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.540417 13533 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
W20260620 12:27:26.541886 13533 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36591 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:27:26.542016 13533 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.542196 13533 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:26.543005 13533 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.543187 13533 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
W20260620 12:27:26.543915 13533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:27:26.713964 13536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:26.714372 13536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:26.714452 13536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:26.721259 13536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:26.721387 13536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:26.721514 13536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:27:26.730062 13536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34735
--tserver_master_addrs=127.0.98.62:36591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:26.731642 13536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:26.733032 13536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:26.744439 13543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:26.744710 13536 server_base.cc:1034] running on GCE node
W20260620 12:27:26.744479 13542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:26.745241 13545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:26.745644 13536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:26.746369 13536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:26.747548 13536 hybrid_clock.cc:648] HybridClock initialized: now 1781958446747541 us; error 66 us; skew 500 ppm
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:26.750530 13536 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:27:26.751608 13536 webserver.cc:460] Webserver started at http://127.0.98.3:34735/ using document root <none> and password file <none>
I20260620 12:27:26.752164 13536 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:26.752269 13536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:26.777307 13536 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 12:27:26.780136 13553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.782253 13536 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:27:26.782380 13536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "08c5ef436fb1430198805b742bfce777"
format_stamp: "Formatted at 2026-06-20 12:27:25 on dist-test-slave-bwpp"
server_key: "034e8e195f5af33c31030246153017d9"
server_key_iv: "cffa0e481c2b94421d1262afb75722a0"
server_key_version: "kuduclusterkey@0"
I20260620 12:27:26.783239 13536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:26.799566 13536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:26.803259 13536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:26.803457 13536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:26.804157 13536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:26.805159 13536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:26.805224 13536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.805284 13536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:26.805318 13536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:26.815805 13536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43195
I20260620 12:27:26.817159 13536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:27:26.817402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13536
I20260620 12:27:26.819582 13666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43195 every 8 connection(s)
I20260620 12:27:26.827528 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:26.818197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.0.98.62:36591)
0620 12:27:26.818368 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:26.818373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:26.818725 (+ 352us) server_negotiation.cc:407] Connection header received
0620 12:27:26.818950 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:26.818953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:26.819018 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:26.819074 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:26.820109 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.820647 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.821723 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.822017 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.822126 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:26.823856 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:26.823861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:26.823864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:26.823888 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:26.825748 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.826165 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.826171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.826173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.826260 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.826536 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.826541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.826543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.826824 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:26.827029 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:26.827243 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 12:27:26.827361 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 12:27:26 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:26.835500 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:26.822003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50109 (local address 127.0.98.62:36591)
0620 12:27:26.823958 (+ 1955us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:26.823967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:26.823991 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:27:26.824025 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:26.824028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:26.824077 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:26.824173 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:26.825711 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.826609 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.828755 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.829177 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.829287 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:26.831840 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:26.831843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:26.831846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:26.831868 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:26.833454 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.833951 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.833957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.833959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.834046 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:26.834396 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:26.834400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:26.834401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:26.834573 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:26.834699 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:26.835204 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 12:27:26.835321 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1879,"thread_start_us":83,"threads_started":1}
I20260620 12:27:26.836335 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:26.820850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36591 (local address 127.0.98.3:50109)
0620 12:27:26.821410 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:26.821463 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:26.823251 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:26.824459 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:26.824472 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:26.824812 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:26.825528 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:26.825551 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.826826 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.826830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:26.828342 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:26.828352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:26.829457 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:26.829461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:26.829658 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:26.830212 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:26.830227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:26.831668 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:26.833574 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:26.833579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:26.833594 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:26.833846 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:26.834139 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:26.834144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:26.834146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:26.834289 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:26.834698 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:26.834704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:26.834964 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 12:27:26.835153 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:27:26.835450 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":163,"threads_started":1}
I20260620 12:27:26.837800 13667 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36591
I20260620 12:27:26.838048 13667 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.838549 13667 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
W20260620 12:27:26.840345 13667 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36591 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:27:26.840471 13667 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.840649 13667 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:26.841526 13667 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:26.841691 13667 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
W20260620 12:27:26.842507 13667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 12:27:27 dist-test-slave-bwpp krb5kdc[12205](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:27.315037 13399 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:27.315347 13399 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:27.316519 13200 ts_manager.cc:194] Registered new tserver with Master: cd09341189494edf8399b5f788d0dafc (127.0.98.1:46629)
I20260620 12:27:27.318274 13200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35005
I20260620 12:27:27.544162 13533 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:27.544445 13533 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:27.545244 13200 ts_manager.cc:194] Registered new tserver with Master: b2ed778160174865b95281502b449d46 (127.0.98.2:41873)
I20260620 12:27:27.546041 13200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46447
Jun 20 12:27:27 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:27.842741 13667 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:27.843034 13667 heartbeater.cc:511] Master 127.0.98.62:36591 requested a full tablet report, sending...
I20260620 12:27:27.843803 13201 ts_manager.cc:194] Registered new tserver with Master: 08c5ef436fb1430198805b742bfce777 (127.0.98.3:43195)
I20260620 12:27:27.844653 13201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50109
I20260620 12:27:27.850310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:27:27.861909 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.851976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.0.98.62:36591)
0620 12:27:27.852222 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:27.852227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:27.852485 (+ 258us) server_negotiation.cc:407] Connection header received
0620 12:27:27.852609 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:27.852612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:27.852670 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:27.852766 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:27.853729 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.854562 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.855750 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.856082 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.856167 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.857816 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:27.857820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:27.857823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:27.857847 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:27.859590 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:27.860132 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:27.860136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:27.860138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:27.860221 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:27.860486 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:27.860489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:27.860491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:27.860674 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:27.860800 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:27.861624 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 12:27:27.861739 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260620 12:27:27.865959 13201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:27:27.870613 13201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:27:27.896656 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.888885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43195 (local address 127.0.0.1:47868)
0620 12:27:27.890127 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:27.890143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:27.890237 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:27.891413 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:27.891417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:27.891439 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:27.891558 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.891565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.893023 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.893028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.894528 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.894540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.896161 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.896165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.896262 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.896299 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 12:27:27.896401 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 12:27:27.896505 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":91,"thread_start_us":76,"threads_started":1}
I20260620 12:27:27.896934 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.889553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.0.98.1:46629)
0620 12:27:27.889858 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:27.889866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:27.889890 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:27:27.889967 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:27.889973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:27.890357 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:27.890509 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:27.890858 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.892926 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.895322 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.896078 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.896174 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.896309 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:27.896720 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 12:27:27.896809 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260620 12:27:27.897223 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.888623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46629 (local address 127.0.0.1:36138)
0620 12:27:27.889352 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:27.889364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:27.889440 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:27.890524 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:27.890530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:27.890554 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:27.890718 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.890725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.893061 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.893065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.894540 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.894552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.896370 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.896373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.896456 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.896478 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:27:27.896542 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 12:27:27.896604 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":24,"thread_start_us":74,"threads_started":1}
Jun 20 12:27:27 dist-test-slave-bwpp krb5kdc[12205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:27:27.900883 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.889871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.0.98.3:43195)
0620 12:27:27.890817 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:27.890823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:27.890847 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:27:27.890920 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:27.890925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:27.891121 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:27.891260 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:27.891703 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.892874 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.894704 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.895837 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.895945 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.896101 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:27.900640 (+ 4539us) server_negotiation.cc:299] Negotiation successful
0620 12:27:27.900739 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":126,"threads_started":1}
I20260620 12:27:27.900775 13333 tablet_service.cc:1467] Processing CreateTablet for tablet 0f44d74bba1d4154a14c45196b47da69 (DEFAULT_TABLE table=test-table [id=e8a10ed3c2374633a0d2d7a85d950734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:27.901943 13333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44d74bba1d4154a14c45196b47da69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:27.903555 13591 tablet_service.cc:1467] Processing CreateTablet for tablet 0f44d74bba1d4154a14c45196b47da69 (DEFAULT_TABLE table=test-table [id=e8a10ed3c2374633a0d2d7a85d950734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:27.904562 13591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44d74bba1d4154a14c45196b47da69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:27.908520 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.887511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41873 (local address 127.0.0.1:53944)
0620 12:27:27.888954 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:27.889017 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:27.889142 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:27.895474 (+ 6332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:27.895477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:27.895494 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:27.895581 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.895587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.897039 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.897043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.898214 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.898225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.908122 (+ 9897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.908126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.908214 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.908235 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:27:27.908316 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:27:27.908389 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"thread_start_us":78,"threads_started":1}
I20260620 12:27:27.913528 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.888257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.0.98.2:41873)
0620 12:27:27.895069 (+ 6812us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:27.895075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:27.895098 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:27:27.895169 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:27.895174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:27.895348 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:27.895508 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:27.895669 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.896736 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.907212 (+ 10476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.908003 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.908126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.908264 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:27.913253 (+ 4989us) server_negotiation.cc:299] Negotiation successful
0620 12:27:27.913371 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6710,"thread_start_us":70,"threads_started":1}
I20260620 12:27:27.917282 13467 tablet_service.cc:1467] Processing CreateTablet for tablet 0f44d74bba1d4154a14c45196b47da69 (DEFAULT_TABLE table=test-table [id=e8a10ed3c2374633a0d2d7a85d950734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:27.918344 13467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44d74bba1d4154a14c45196b47da69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:27.929625 13701 tablet_bootstrap.cc:492] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc: Bootstrap starting.
I20260620 12:27:27.932088 13701 tablet_bootstrap.cc:654] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:27.932536 13703 tablet_bootstrap.cc:492] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: Bootstrap starting.
I20260620 12:27:27.932979 13701 log.cc:824] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:27.933735 13702 tablet_bootstrap.cc:492] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Bootstrap starting.
I20260620 12:27:27.934623 13703 tablet_bootstrap.cc:654] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:27.935719 13703 log.cc:824] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:27.935855 13702 tablet_bootstrap.cc:654] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:27.936806 13702 log.cc:824] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:27.938372 13701 tablet_bootstrap.cc:492] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc: No bootstrap required, opened a new log
I20260620 12:27:27.939190 13702 tablet_bootstrap.cc:492] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: No bootstrap required, opened a new log
I20260620 12:27:27.939388 13702 ts_tablet_manager.cc:1397] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 12:27:27.939818 13703 tablet_bootstrap.cc:492] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: No bootstrap required, opened a new log
I20260620 12:27:27.940062 13703 ts_tablet_manager.cc:1397] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:27:27.944033 13703 raft_consensus.cc:348] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.944033 13702 raft_consensus.cc:348] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.944308 13702 raft_consensus.cc:374] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:27.944308 13703 raft_consensus.cc:374] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:27.944360 13702 raft_consensus.cc:729] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c5ef436fb1430198805b742bfce777, State: Initialized, Role: FOLLOWER
I20260620 12:27:27.944360 13703 raft_consensus.cc:729] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ed778160174865b95281502b449d46, State: Initialized, Role: FOLLOWER
I20260620 12:27:27.944751 13701 ts_tablet_manager.cc:1397] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260620 12:27:27.944864 13702 consensus_queue.cc:260] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.945953 13702 ts_tablet_manager.cc:1428] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:27:27.946309 13667 heartbeater.cc:503] Master 127.0.98.62:36591 was elected leader, sending a full tablet report...
I20260620 12:27:27.946518 13703 consensus_queue.cc:260] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.947487 13703 ts_tablet_manager.cc:1428] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 12:27:27.947856 13533 heartbeater.cc:503] Master 127.0.98.62:36591 was elected leader, sending a full tablet report...
I20260620 12:27:27.951009 13701 raft_consensus.cc:348] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.951239 13701 raft_consensus.cc:374] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:27.951293 13701 raft_consensus.cc:729] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd09341189494edf8399b5f788d0dafc, State: Initialized, Role: FOLLOWER
I20260620 12:27:27.951706 13701 consensus_queue.cc:260] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.952431 13701 ts_tablet_manager.cc:1428] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:27:27.952756 13399 heartbeater.cc:503] Master 127.0.98.62:36591 was elected leader, sending a full tablet report...
I20260620 12:27:27.971153 13707 raft_consensus.cc:482] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:27.971400 13707 raft_consensus.cc:504] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.972921 13707 leader_election.cc:288] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2ed778160174865b95281502b449d46 (127.0.98.2:41873), cd09341189494edf8399b5f788d0dafc (127.0.98.1:46629)
I20260620 12:27:27.983709 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.973488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39563 (local address 127.0.98.2:41873)
0620 12:27:27.973613 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:27.973618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:27.978710 (+ 5092us) server_negotiation.cc:407] Connection header received
0620 12:27:27.978950 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:27.978953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:27.979022 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:27.979063 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:27.979749 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.980340 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.982150 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.982791 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.982899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.982969 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:27.983472 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 12:27:27.983558 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:27:27.984323 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.973095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41873 (local address 127.0.98.3:39563)
0620 12:27:27.978589 (+ 5494us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:27.978606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:27.978808 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:27.979274 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:27.979277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:27.979295 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:27.979449 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.979456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.980666 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.980673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.981997 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.982009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.983111 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.983115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.983309 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.983336 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:27:27.984042 (+ 706us) client_negotiation.cc:240] Negotiation successful
0620 12:27:27.984173 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5419,"thread_start_us":106,"threads_started":1}
I20260620 12:27:27.985071 13487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44d74bba1d4154a14c45196b47da69" candidate_uuid: "08c5ef436fb1430198805b742bfce777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2ed778160174865b95281502b449d46" is_pre_election: true
I20260620 12:27:27.985426 13487 raft_consensus.cc:2454] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c5ef436fb1430198805b742bfce777 in term 0.
I20260620 12:27:27.985970 13555 leader_election.cc:302] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c5ef436fb1430198805b742bfce777, b2ed778160174865b95281502b449d46; no voters:
I20260620 12:27:27.986253 13707 raft_consensus.cc:2781] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:27:27.986352 13707 raft_consensus.cc:482] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:27:27.986393 13707 raft_consensus.cc:3037] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:27.988286 13707 raft_consensus.cc:504] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.989045 13487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44d74bba1d4154a14c45196b47da69" candidate_uuid: "08c5ef436fb1430198805b742bfce777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2ed778160174865b95281502b449d46"
I20260620 12:27:27.989176 13487 raft_consensus.cc:3037] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:27.989595 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.977739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51423 (local address 127.0.98.1:46629)
0620 12:27:27.977859 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:27.977863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:27.984913 (+ 7050us) server_negotiation.cc:407] Connection header received
0620 12:27:27.985189 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:27.985193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:27.985250 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:27.985298 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:27.985667 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.986401 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.988263 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.988855 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.988929 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.989025 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:27.989390 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 12:27:27.989476 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:27:27.989897 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:27.977253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46629 (local address 127.0.98.3:51423)
0620 12:27:27.984833 (+ 7580us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:27.984848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:27.985015 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:27.985437 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:27.985440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:27.985458 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:27.985532 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.985539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.986550 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.986553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.988115 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:27.988129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:27.989152 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:27.989156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:27.989242 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:27.989266 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:27:27.989716 (+ 450us) client_negotiation.cc:240] Negotiation successful
0620 12:27:27.989783 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7519,"thread_start_us":97,"threads_started":1}
I20260620 12:27:27.990525 13487 raft_consensus.cc:2454] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c5ef436fb1430198805b742bfce777 in term 1.
I20260620 12:27:27.990581 13343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44d74bba1d4154a14c45196b47da69" candidate_uuid: "08c5ef436fb1430198805b742bfce777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd09341189494edf8399b5f788d0dafc" is_pre_election: true
I20260620 12:27:27.990849 13555 leader_election.cc:302] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c5ef436fb1430198805b742bfce777, b2ed778160174865b95281502b449d46; no voters:
I20260620 12:27:27.990883 13343 raft_consensus.cc:2454] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c5ef436fb1430198805b742bfce777 in term 0.
I20260620 12:27:27.991640 13707 leader_election.cc:288] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [CANDIDATE]: Term 1 election: Requested vote from peers b2ed778160174865b95281502b449d46 (127.0.98.2:41873), cd09341189494edf8399b5f788d0dafc (127.0.98.1:46629)
I20260620 12:27:27.991768 13707 raft_consensus.cc:2781] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:27.991863 13707 raft_consensus.cc:686] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 1 LEADER]: Becoming Leader. State: Replica: 08c5ef436fb1430198805b742bfce777, State: Running, Role: LEADER
I20260620 12:27:27.991968 13343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44d74bba1d4154a14c45196b47da69" candidate_uuid: "08c5ef436fb1430198805b742bfce777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd09341189494edf8399b5f788d0dafc"
I20260620 12:27:27.992067 13343 raft_consensus.cc:3037] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:27.992244 13707 consensus_queue.cc:237] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } }
I20260620 12:27:27.993296 13343 raft_consensus.cc:2454] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c5ef436fb1430198805b742bfce777 in term 1.
I20260620 12:27:27.995883 13201 catalog_manager.cc:5495] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08c5ef436fb1430198805b742bfce777 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "08c5ef436fb1430198805b742bfce777" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08c5ef436fb1430198805b742bfce777" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43195 } health_report { overall_health: HEALTHY } } }
W20260620 12:27:28.022122 13534 tablet.cc:2369] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:27:28.033727 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:28.028468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.0.98.3:43195)
0620 12:27:28.028613 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:28.028619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:28.028633 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:27:28.028798 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:28.028802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:28.028878 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:28.028990 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:27:28.029408 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:28.030248 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:28.031541 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:28.031883 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:28.031987 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:28.032404 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:27:28.032985 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:27:28.033102 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:28.033525 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 12:27:28.033589 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 12:27:28.034623 13400 tablet.cc:2369] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:27:28.041185 13343 raft_consensus.cc:1261] T 0f44d74bba1d4154a14c45196b47da69 P cd09341189494edf8399b5f788d0dafc [term 1 FOLLOWER]: Refusing update from remote peer 08c5ef436fb1430198805b742bfce777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:28.042040 13713 consensus_queue.cc:1035] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd09341189494edf8399b5f788d0dafc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:28.048691 13487 raft_consensus.cc:1261] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 1 FOLLOWER]: Refusing update from remote peer 08c5ef436fb1430198805b742bfce777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:28.049453 13713 consensus_queue.cc:1035] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ed778160174865b95281502b449d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:28.063346 13716 mvcc.cc:205] Tried to move back new op lower bound from 7298901803167895552 to 7298901802979438592. Current Snapshot: MvccSnapshot[applied={T|T < 7298901803167895552}]
I20260620 12:27:28.063764 13719 mvcc.cc:205] Tried to move back new op lower bound from 7298901803167895552 to 7298901802979438592. Current Snapshot: MvccSnapshot[applied={T|T < 7298901803167895552}]
I20260620 12:27:28.072196 13717 mvcc.cc:205] Tried to move back new op lower bound from 7298901803167895552 to 7298901802979438592. Current Snapshot: MvccSnapshot[applied={T|T < 7298901803167895552}]
I20260620 12:27:28.073315 13201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:28.073467 13201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:28.076100 13201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5022031f963149df99dd39b7bc1fe868: Sending DeleteTablet for 3 replicas of tablet 0f44d74bba1d4154a14c45196b47da69
I20260620 12:27:28.076967 13467 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f44d74bba1d4154a14c45196b47da69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53944
I20260620 12:27:28.077857 13333 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f44d74bba1d4154a14c45196b47da69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36138
I20260620 12:27:28.078583 13591 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f44d74bba1d4154a14c45196b47da69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47868
I20260620 12:27:28.079550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13239
I20260620 12:27:28.082310 13727 tablet_replica.cc:330] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: stopping tablet replica
I20260620 12:27:28.083998 13727 raft_consensus.cc:2229] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:27:28.084445 13727 raft_consensus.cc:2258] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:28.088270 13725 tablet_replica.cc:330] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: stopping tablet replica
I20260620 12:27:28.088830 13727 ts_tablet_manager.cc:1905] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:28.089066 13725 raft_consensus.cc:2229] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:28.089401 13725 raft_consensus.cc:2258] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:28.093146 13725 ts_tablet_manager.cc:1905] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:28.095733 13725 ts_tablet_manager.cc:1918] T 0f44d74bba1d4154a14c45196b47da69 P b2ed778160174865b95281502b449d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:28.098097 13727 ts_tablet_manager.cc:1918] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:28.098248 13727 log.cc:1196] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/0f44d74bba1d4154a14c45196b47da69
W20260620 12:27:28.098853 13185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46629 (error 108)
I20260620 12:27:28.099774 13727 ts_tablet_manager.cc:1939] T 0f44d74bba1d4154a14c45196b47da69 P 08c5ef436fb1430198805b742bfce777: Deleting consensus metadata
I20260620 12:27:28.100023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13403
I20260620 12:27:28.102090 13185 catalog_manager.cc:4841] TS 08c5ef436fb1430198805b742bfce777 (127.0.98.3:43195): tablet 0f44d74bba1d4154a14c45196b47da69 (table test-table [id=e8a10ed3c2374633a0d2d7a85d950734]) successfully deleted
I20260620 12:27:28.107527 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:28.107198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46629 (local address 127.0.0.1:36140)
0620 12:27:28.107372 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:28.107440 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 12:27:28.107772 13185 catalog_manager.cc:4584] TS cd09341189494edf8399b5f788d0dafc (127.0.98.1:46629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f44d74bba1d4154a14c45196b47da69: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46629: connect: Connection refused (error 111)
W20260620 12:27:28.109889 13697 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:27:28.109348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41873 (local address 127.0.0.1:53960)
0620 12:27:28.109489 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:28.109503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:28.109588 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:28.109805 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 12:27:28.110546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13536
I20260620 12:27:28.117867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13168
I20260620 12:27:28.124841 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 12:27:28.458912 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 12:27:28.459167 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 12:27:28.803658 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 12:27:28.803 UTC [12235] LOG: received smart shutdown request
2026-06-20 12:27:28.807 UTC [12235] LOG: background worker "logical replication launcher" (PID 12244) exited with exit code 1
2026-06-20 12:27:28.808 UTC [12239] LOG: shutting down
2026-06-20 12:27:28.965 UTC [12235] LOG: database system is shut down
2026-06-20T12:27:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47907 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:27:29 dist-test-slave-bwpp krb5kdc[13737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:27:29 dist-test-slave-bwpp krb5kdc[13737](info): set up 2 sockets
Jun 20 12:27:29 dist-test-slave-bwpp krb5kdc[13737](info): commencing operation
krb5kdc: starting...
W20260620 12:27:30.148487 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:27:30.269104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40639,127.0.98.61:43845,127.0.98.60:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:30.377676 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:30.377924 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:30.377962 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:30.381834 13750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:30.381886 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:30.381906 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:30.381922 13750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:30.381937 13750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:30.387027 13750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40639,127.0.98.61:43845,127.0.98.60:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:30.387933 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:30.388875 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:30.397135 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:30.397208 13756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:30.397177 13750 server_base.cc:1034] running on GCE node
W20260620 12:27:30.397135 13755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:30.397953 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:30.399096 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:30.400301 13750 hybrid_clock.cc:648] HybridClock initialized: now 1781958450400280 us; error 34 us; skew 500 ppm
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:30.402958 13750 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:30.403712 13750 webserver.cc:460] Webserver started at http://127.0.98.62:44803/ using document root <none> and password file <none>
I20260620 12:27:30.404228 13750 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:30.404285 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:30.404440 13750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:30.406281 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "92d9fd04006949d5998a43e60bd5c1bd"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "0da985a5064b460cf6bbc7c03128b247"
server_key_iv: "66314c48465459727bd00c55c42f96f5"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.406839 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "92d9fd04006949d5998a43e60bd5c1bd"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "0da985a5064b460cf6bbc7c03128b247"
server_key_iv: "66314c48465459727bd00c55c42f96f5"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.410281 13750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:27:30.412554 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:30.413687 13750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:30.413789 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "92d9fd04006949d5998a43e60bd5c1bd"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "0da985a5064b460cf6bbc7c03128b247"
server_key_iv: "66314c48465459727bd00c55c42f96f5"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.413865 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:30.428540 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:30.431125 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:30.431264 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:30.437827 13750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40639
I20260620 12:27:30.437881 13817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40639 every 8 connection(s)
I20260620 12:27:30.439047 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:30.442226 13818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:30.444666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13750
I20260620 12:27:30.444795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:27:30.445127 392 external_mini_cluster.cc:1428] Setting key 2783af8f2c616c26dc91edea1b02986d
I20260620 12:27:30.446421 13818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:30.462070 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.448819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.98.62:40639)
0620 12:27:30.449355 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.449368 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.449414 (+ 46us) server_negotiation.cc:407] Connection header received
0620 12:27:30.450439 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.450447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.450740 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.450960 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.451702 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.453149 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.454159 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.454565 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.454744 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.457427 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.457433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.457442 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.457461 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.459171 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.459656 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.459663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.459670 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.459762 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.460061 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.460066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.460068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.460400 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.460537 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.460881 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.461109 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":406,"spinlock_wait_cycles":17280,"thread_start_us":137,"threads_started":1}
I20260620 12:27:30.462070 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.448819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40639 (local address 127.0.0.1:53884)
0620 12:27:30.449262 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.449311 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.450122 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.450956 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.450961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.451258 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.451552 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.451564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.453288 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.453290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.454050 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.454059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.454779 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.454782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.454902 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.455408 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.455423 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.457232 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.459304 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.459312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.459321 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.459537 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.459854 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.459859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.459861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.459969 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.460512 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.460517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.460653 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.460819 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.461158 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":128,"thread_start_us":138,"threads_started":1}
I20260620 12:27:30.464337 13818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:27:30.464996 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.464751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41732)
0620 12:27:30.464867 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.464938 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 12:27:30.465183 13768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
I20260620 12:27:30.465955 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.465748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41744)
0620 12:27:30.465854 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.465906 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 12:27:30.466092 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43845: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 12:27:30.469524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43845
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40639,127.0.98.61:43845,127.0.98.60:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:30.515362 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } attempt: 1
I20260620 12:27:30.516249 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.515954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41748)
0620 12:27:30.516110 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.516174 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:27:30.517019 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.516791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41754)
0620 12:27:30.516887 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.516955 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 12:27:30.517163 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43845: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
W20260620 12:27:30.578011 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:30.578305 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:30.578383 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:30.582433 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:30.582511 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:30.582551 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:30.582605 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:30.582680 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:30.585332 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } attempt: 2
I20260620 12:27:30.586148 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.585863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41762)
0620 12:27:30.586010 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.586074 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 12:27:30.586966 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.586739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41774)
0620 12:27:30.586852 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.586910 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 12:27:30.587110 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43845: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43845: connect: Connection refused (error 111)
I20260620 12:27:30.588440 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40639,127.0.98.61:43845,127.0.98.60:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:30.589430 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:30.590425 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:30.598678 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:30.598572 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:30.598701 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:30.598899 13825 server_base.cc:1034] running on GCE node
I20260620 12:27:30.599421 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:30.600385 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:30.601541 13825 hybrid_clock.cc:648] HybridClock initialized: now 1781958450601514 us; error 44 us; skew 500 ppm
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:30.604238 13825 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 12:27:30.605125 13825 webserver.cc:460] Webserver started at http://127.0.98.61:42885/ using document root <none> and password file <none>
I20260620 12:27:30.605612 13825 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:30.605670 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:30.605820 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:30.608240 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data/instance:
uuid: "93deb4c409f94f20a0e2d211bf153027"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "98336c801ef475d591031eb3b09b7ca9"
server_key_iv: "80f2b0d7910a9cfdc8ffd9afa857d79b"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.608736 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/wal/instance:
uuid: "93deb4c409f94f20a0e2d211bf153027"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "98336c801ef475d591031eb3b09b7ca9"
server_key_iv: "80f2b0d7910a9cfdc8ffd9afa857d79b"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.612262 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 12:27:30.614533 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:30.615743 13825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:27:30.615840 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/wal
uuid: "93deb4c409f94f20a0e2d211bf153027"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "98336c801ef475d591031eb3b09b7ca9"
server_key_iv: "80f2b0d7910a9cfdc8ffd9afa857d79b"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.615923 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:30.639290 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:30.642203 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:30.642345 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:30.649349 13825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43845
I20260620 12:27:30.649376 13892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43845 every 8 connection(s)
I20260620 12:27:30.650274 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:27:30.652907 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:30.655232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13825
I20260620 12:27:30.655324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-1/wal/instance
I20260620 12:27:30.655629 392 external_mini_cluster.cc:1428] Setting key b21946aa34de5fffbb2934999ab15683
I20260620 12:27:30.656731 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:30.672344 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.658560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.98.62:40639)
0620 12:27:30.658722 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.658726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.659675 (+ 949us) server_negotiation.cc:407] Connection header received
0620 12:27:30.660584 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.660587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.660647 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.660752 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.662173 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.662840 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.664318 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.664579 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.664703 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.667915 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.667923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.667928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.667997 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.669925 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.670446 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.670454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.670459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.670561 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.670953 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.670960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.670965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.671246 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.671374 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.671903 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.672065 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:27:30.672904 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.659025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40639 (local address 127.0.0.1:53898)
0620 12:27:30.659527 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.659567 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.660374 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.660923 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.660930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.661482 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.661996 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.662010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.662977 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.662982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.664174 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.664184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.664742 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.664747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.664983 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.665685 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.665703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.667678 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.670052 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.670057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.670070 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.670336 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.670713 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.670715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.670717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.670850 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.671357 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.671362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.671607 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.671760 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.672022 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260620 12:27:30.674526 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:27:30.678309 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } attempt: 3
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 12:27:30.683987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37507
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40639,127.0.98.61:43845,127.0.98.60:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:30.684216 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.675128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41776)
0620 12:27:30.675274 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.675284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.675344 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.675895 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.675898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.676076 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.676144 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.676149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.677101 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.677107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.677632 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.677639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.678199 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.678202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.678291 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.679064 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.679083 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.680301 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.682410 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.682416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.682421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.682708 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.683036 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.683042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.683047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.683136 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.683663 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.683668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.683777 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.683905 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.684035 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:27:30.684600 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.675249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.0.98.61:43845)
0620 12:27:30.675573 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.675581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.675598 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:27:30.675629 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.675633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.675777 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.675921 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.676258 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.676994 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.677744 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.678090 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.678204 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.680430 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.680440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.680453 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.680478 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.682291 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.682827 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.682835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.682840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.682931 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.683228 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.683234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.683239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.683535 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.683675 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.683990 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.684114 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":132,"threads_started":1}
I20260620 12:27:30.686619 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:27:30.687409 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.687141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52894)
0620 12:27:30.687249 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.687315 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 12:27:30.687587 13843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
I20260620 12:27:30.688397 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.688188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52906)
0620 12:27:30.688286 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.688334 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 12:27:30.688470 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.678788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41784)
0620 12:27:30.678909 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.678924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.679006 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.679410 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.679413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.679624 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.680108 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.680117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.681316 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.681321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.682009 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.682019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.682635 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.682642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.682827 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.683279 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.683290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.684675 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.686694 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.686700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.686705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.686940 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.687278 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.687284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.687289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.687397 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.687857 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.687863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.687975 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.688061 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.688209 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 12:27:30.688536 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
I20260620 12:27:30.689481 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.678836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.98.61:43845)
0620 12:27:30.679187 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.679196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.679222 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:27:30.679249 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.679251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.679284 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.679395 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.680209 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.681095 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.682128 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.682527 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.682598 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.684848 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.684854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.684860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.684890 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.686522 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.687086 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.687089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.687090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.687170 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.687513 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.687518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.687523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.687744 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.689019 (+ 1275us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.689106 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.689234 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":90,"threads_started":1}
I20260620 12:27:30.690109 13818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:27:30.690907 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.690558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52914)
0620 12:27:30.690757 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.690819 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 12:27:30.691658 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.691429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52922)
0620 12:27:30.691567 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.691609 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 12:27:30.691968 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
I20260620 12:27:30.723140 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } attempt: 1
I20260620 12:27:30.723984 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.723680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52932)
0620 12:27:30.723840 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.723902 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 12:27:30.724716 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.724536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52938)
0620 12:27:30.724625 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.724667 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 12:27:30.724884 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
I20260620 12:27:30.737804 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } attempt: 1
I20260620 12:27:30.738569 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.738303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52942)
0620 12:27:30.738427 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.738490 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 12:27:30.739336 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.739127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52954)
0620 12:27:30.739230 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.739281 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 12:27:30.739470 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
W20260620 12:27:30.796868 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:30.797111 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:30.797166 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 12:27:30.800073 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } attempt: 2
I20260620 12:27:30.800909 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.800630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52968)
0620 12:27:30.800773 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.800835 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 12:27:30.801594 13900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:30.801659 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:30.801678 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260620 12:27:30.801640 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.801432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52982)
0620 12:27:30.801528 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.801569 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 12:27:30.801707 13900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:30.801748 13900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:27:30.801846 13818 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
I20260620 12:27:30.807154 13900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40639,127.0.98.61:43845,127.0.98.60:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:30.807612 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } attempt: 2
I20260620 12:27:30.808211 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:30.808300 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.808035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:52998)
0620 12:27:30.808153 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.808210 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 12:27:30.809001 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.808814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:53002)
0620 12:27:30.808901 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.808943 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 12:27:30.809144 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37507: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37507: connect: Connection refused (error 111)
I20260620 12:27:30.809494 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:30.817843 13909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:30.817844 13907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:30.817866 13906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:30.818099 13900 server_base.cc:1034] running on GCE node
I20260620 12:27:30.818890 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:30.820097 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:30.821257 13900 hybrid_clock.cc:648] HybridClock initialized: now 1781958450821239 us; error 32 us; skew 500 ppm
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:30.824087 13900 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 12:27:30.825016 13900 webserver.cc:460] Webserver started at http://127.0.98.60:36609/ using document root <none> and password file <none>
I20260620 12:27:30.825515 13900 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:30.825570 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:30.825726 13900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:30.827715 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data/instance:
uuid: "cc95ff5c9066428fa768f7c2925a4953"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "e735f460a1544b29a244c562e41144dd"
server_key_iv: "6eafe3a22b35d14fec0861ee3c3e76df"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.828202 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/wal/instance:
uuid: "cc95ff5c9066428fa768f7c2925a4953"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "e735f460a1544b29a244c562e41144dd"
server_key_iv: "6eafe3a22b35d14fec0861ee3c3e76df"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.831744 13900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:27:30.833902 13916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:30.835150 13900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:27:30.835306 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/wal
uuid: "cc95ff5c9066428fa768f7c2925a4953"
format_stamp: "Formatted at 2026-06-20 12:27:30 on dist-test-slave-bwpp"
server_key: "e735f460a1544b29a244c562e41144dd"
server_key_iv: "6eafe3a22b35d14fec0861ee3c3e76df"
server_key_version: "encryptionkey@0"
I20260620 12:27:30.835412 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:30.871433 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:30.874338 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:30.874547 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:30.881228 13900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37507
I20260620 12:27:30.881256 13968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37507 every 8 connection(s)
I20260620 12:27:30.882158 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:27:30.884794 13969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:30.888504 13969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:27:30.891825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 13900
I20260620 12:27:30.891950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781958290914605-392-0/minicluster-data/master-2/wal/instance
I20260620 12:27:30.892266 392 external_mini_cluster.cc:1428] Setting key cd1fde4a8b7e6103886eef48ce3b6ef7
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:27:30.900314 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } attempt: 3
I20260620 12:27:30.901049 13818 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } attempt: 3
I20260620 12:27:30.903910 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.890345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.98.62:40639)
0620 12:27:30.890521 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.890526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.891379 (+ 853us) server_negotiation.cc:407] Connection header received
0620 12:27:30.892615 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.892620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.892682 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.892773 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.894561 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.895140 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.896222 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.896476 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.896552 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.899339 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.899343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.899345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.899377 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.901197 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.902135 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.902139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.902141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.902226 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.902612 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.902615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.902617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.902882 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.902955 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.903564 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.903708 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:27:30.905441 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.890804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40639 (local address 127.0.0.1:53908)
0620 12:27:30.891230 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.891271 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.892339 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.893355 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.893367 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.893844 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.894406 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.894426 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.895245 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.895249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.896089 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.896098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.896607 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.896611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.896823 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.897568 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.897583 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.899194 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.901334 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.901345 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.901358 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.902015 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.902342 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.902345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.902347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.902496 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.903044 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.903049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.903296 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.904087 (+ 791us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.904402 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":86,"spinlock_wait_cycles":4736,"thread_start_us":100,"threads_started":1}
I20260620 12:27:30.905438 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.892936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.0.98.60:37507)
0620 12:27:30.893231 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.893241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.893266 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:27:30.893387 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.893393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.893576 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.893762 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.894445 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.895460 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.896287 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.896748 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.896859 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.899804 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.899814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.899838 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.899867 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.901853 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.902349 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.902355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.902357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.902497 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.902836 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.902840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.902841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.903296 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.904936 (+ 1640us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.905036 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.905165 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":205,"spinlock_wait_cycles":15232,"thread_start_us":143,"threads_started":1}
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:27:30.907068 13969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:27:30.913371 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.901449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:53026)
0620 12:27:30.901557 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.901570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.901641 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.902128 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.902131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.902308 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.902404 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.902410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.903706 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.903709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.904518 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.904527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.905302 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.905306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.905386 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.906011 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.906026 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.908332 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.911124 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.911127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.911129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.911660 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.912265 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.912269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.912271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.912365 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.912838 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.912842 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.912935 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.913063 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.913202 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 12:27:30.913427 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.901571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.0.98.60:37507)
0620 12:27:30.901927 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.901934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.901944 (+ 10us) server_negotiation.cc:407] Connection header received
0620 12:27:30.901967 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.901969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.901998 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.902517 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.902717 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.903575 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.904664 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.905173 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.905584 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.908462 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.908466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.908468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.908491 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.911000 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.911777 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.911783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.911785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.912164 (+ 379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.912472 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.912476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.912478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.912726 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.912830 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.913123 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.913236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":303,"spinlock_wait_cycles":10624,"thread_start_us":102,"threads_started":1}
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:30.913689 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.900819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:53022)
0620 12:27:30.900987 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.901003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.901091 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.902134 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.902136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.902309 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.902415 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.902422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.903706 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.903709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.904518 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.904527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.905302 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.905306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.905393 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.906011 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.906026 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.908740 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.911008 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.911011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.911013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.911282 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.911602 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.911607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.911613 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.911709 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.913216 (+ 1507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.913220 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.913313 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.913391 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.913526 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 12:27:30.914184 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.901308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.0.98.60:37507)
0620 12:27:30.901883 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.901892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.901905 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:27:30.901941 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.901944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.901988 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.902517 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.902720 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.903575 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.904664 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.905173 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.906115 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.908898 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.908902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.908905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.908931 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.910881 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.911408 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.911413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.911414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.911491 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.911812 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.911818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.911821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.912779 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.913883 (+ 1104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.913956 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.914037 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":485,"spinlock_wait_cycles":15872,"thread_start_us":86,"threads_started":1}
I20260620 12:27:30.917435 13818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd: Bootstrap starting.
I20260620 12:27:30.917742 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: Bootstrap starting.
I20260620 12:27:30.918051 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.907773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.0.98.61:43845)
0620 12:27:30.907898 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.907902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.907915 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:27:30.907946 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.907949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.907994 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.908067 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.908503 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.909260 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.910297 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.910679 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.910750 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.914368 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.914374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.914376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.914405 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.916155 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.916658 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.916662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.916664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.916746 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.917070 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.917074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.917075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.917323 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.917398 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.917755 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.917888 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:27:30.918198 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.907918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.0.98.62:40639)
0620 12:27:30.908015 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.908019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.908107 (+ 88us) server_negotiation.cc:407] Connection header received
0620 12:27:30.908274 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.908277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.908315 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.908367 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.909182 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.909906 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.911009 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.911339 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.911438 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.914194 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.914200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.914203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.914227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.916262 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.916786 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.916790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.916792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.916870 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.917178 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.917181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.917183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.917376 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.917540 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.917957 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.918055 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 12:27:30.918481 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.907682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43845 (local address 127.0.0.1:41788)
0620 12:27:30.907765 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.907779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.907847 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.908091 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.908097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.908267 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.908374 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.908381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.909386 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.909390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.910170 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.910180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.910886 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.910889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.911010 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.911458 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.911473 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.913091 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.916283 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.916289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.916294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.916537 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.916875 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.916878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.916879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.916967 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.917478 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.917480 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.917573 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.918207 (+ 634us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.918333 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"spinlock_wait_cycles":1280}
I20260620 12:27:30.919423 13969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:27:30.921079 13893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:30.921079 13818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:30.921993 13818 log.cc:824] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:30.921983 13893 log.cc:824] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:30.924499 13818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd: No bootstrap required, opened a new log
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 12:27:30 dist-test-slave-bwpp krb5kdc[13737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:27:30.928491 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.919958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.0.98.61:43845)
0620 12:27:30.920060 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.920064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.920205 (+ 141us) server_negotiation.cc:407] Connection header received
0620 12:27:30.920367 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.920370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.920448 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.920515 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.921072 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.921545 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.922345 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.922638 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.922758 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.925113 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.925117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.925120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.925142 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.926816 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.927294 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.927297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.927299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.927380 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.927641 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.927644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.927646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.927855 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.928028 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.928252 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.928355 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:27:30.928494 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: No bootstrap required, opened a new log
I20260620 12:27:30.930579 13818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:30.930837 13818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:30.930907 13818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d9fd04006949d5998a43e60bd5c1bd, State: Initialized, Role: FOLLOWER
I20260620 12:27:30.931420 13818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:30.931687 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:27:30.932097 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.920049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.98.60:37507)
0620 12:27:30.920143 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.920146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.922532 (+ 2386us) server_negotiation.cc:407] Connection header received
0620 12:27:30.922598 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.922601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.922643 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.922741 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.923461 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.924145 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.925084 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.925383 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.925475 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.927629 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.927634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.927640 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.927664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.930111 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.930555 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.930558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.930559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.930622 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.931105 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.931108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.931110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.931328 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.931427 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.931832 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.931940 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 12:27:30.932591 13771 tablet.cc:2369] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:27:30.932924 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.919863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37507 (local address 127.0.0.1:53042)
0620 12:27:30.922444 (+ 2581us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:30.922466 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:30.922539 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:30.923098 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:30.923103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:30.923264 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:30.923346 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.923352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.924263 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.924266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.924950 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:30.924961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.925568 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.925571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:30.925642 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.926018 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:30.926031 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:30.927490 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:30.930224 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.930227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.930229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.930462 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.930747 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:30.930750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:30.930751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:30.931001 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:30.931457 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:30.931460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:30.931537 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 12:27:30.931731 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 12:27:30.932787 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2518}
I20260620 12:27:30.933141 13893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:30.933372 13893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:30.933383 13818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:30.933444 13893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93deb4c409f94f20a0e2d211bf153027, State: Initialized, Role: FOLLOWER
I20260620 12:27:30.933897 13893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:30.935177 13893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:30.936071 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:30.936208 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:27:30.936434 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:30.937071 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:27:30.937152 13969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: Bootstrap starting.
I20260620 12:27:30.940433 13969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:30.941264 13969 log.cc:824] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: Log is configured to *not* fsync() on all Append() calls
W20260620 12:27:30.941793 14008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:27:30.941855 14008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:27:30.938282 13750 master_runner.cc:186] Error getting master registration for 127.0.98.61:43845: OK, instance_id { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" instance_seqno: 1781958450647393 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:27:30.942233 13750 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:27:30.943413 13969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: No bootstrap required, opened a new log
W20260620 12:27:30.944794 13996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:27:30.944868 13996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:27:30.943378 13825 master_runner.cc:186] Error getting master registration for 127.0.98.60:37507: OK, instance_id { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" instance_seqno: 1781958450879280 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:27:30.946575 13825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:27:30.948335 13969 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:30.948599 13969 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:30.948699 13969 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc95ff5c9066428fa768f7c2925a4953, State: Initialized, Role: FOLLOWER
I20260620 12:27:30.949259 13969 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:30.950793 13969 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:30.949853 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:30.951440 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:27:30.955224 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.936156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.0.98.60:37507)
0620 12:27:30.936262 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.936267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.945406 (+ 9139us) server_negotiation.cc:407] Connection header received
0620 12:27:30.945683 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.945686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.945730 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.945779 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.946801 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.947676 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.948834 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.949145 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.949219 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.951119 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.951126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.951130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.951162 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.952992 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.953484 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.953490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.953494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.953587 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.953885 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.953888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.953890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.954101 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.954243 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.954955 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.955067 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 12:27:30.959220 14023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:27:30.959300 14023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:27:30.959438 13900 master_runner.cc:263] No leader master found from master cc95ff5c9066428fa768f7c2925a4953
I20260620 12:27:30.959491 13900 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:27:30.962833 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.934131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.0.98.61:43845)
0620 12:27:30.934808 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.934812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.945199 (+ 10387us) server_negotiation.cc:407] Connection header received
0620 12:27:30.946782 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.946785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.946843 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.946896 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.947824 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.948584 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.951677 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.952283 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.952362 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.958401 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.958408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.958414 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.958444 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.960844 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.961387 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.961393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.961396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.961486 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.961792 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.961796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.961798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.962038 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.962153 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.962507 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.962640 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608}
I20260620 12:27:30.968123 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:30.935783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.98.62:40639)
0620 12:27:30.935882 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:30.935886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:30.950280 (+ 14394us) server_negotiation.cc:407] Connection header received
0620 12:27:30.950455 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:30.950459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:30.950504 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:30.950556 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:30.953009 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.953767 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.957381 (+ 3614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:30.957685 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:30.957762 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:30.964357 (+ 6595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:30.964361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:30.964364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:30.964394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:30.966144 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.966642 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.966645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.966673 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.966763 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:30.967102 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:30.967107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:30.967109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:30.967337 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:30.967452 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:30.967806 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 12:27:30.967953 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:27:31.072929 13978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:31.073132 13978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:31.074198 13978 leader_election.cc:288] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93deb4c409f94f20a0e2d211bf153027 (127.0.98.61:43845), cc95ff5c9066428fa768f7c2925a4953 (127.0.98.60:37507)
I20260620 12:27:31.075071 13867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92d9fd04006949d5998a43e60bd5c1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93deb4c409f94f20a0e2d211bf153027" is_pre_election: true
I20260620 12:27:31.075416 13867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d9fd04006949d5998a43e60bd5c1bd in term 0.
I20260620 12:27:31.075977 13943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92d9fd04006949d5998a43e60bd5c1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc95ff5c9066428fa768f7c2925a4953" is_pre_election: true
I20260620 12:27:31.076308 13943 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d9fd04006949d5998a43e60bd5c1bd in term 0.
I20260620 12:27:31.076817 13768 leader_election.cc:302] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92d9fd04006949d5998a43e60bd5c1bd, cc95ff5c9066428fa768f7c2925a4953; no voters:
I20260620 12:27:31.077350 13978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:27:31.077445 13978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:27:31.077502 13978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:31.078907 13978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:31.079276 13978 leader_election.cc:288] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [CANDIDATE]: Term 1 election: Requested vote from peers 93deb4c409f94f20a0e2d211bf153027 (127.0.98.61:43845), cc95ff5c9066428fa768f7c2925a4953 (127.0.98.60:37507)
I20260620 12:27:31.079879 13943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92d9fd04006949d5998a43e60bd5c1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc95ff5c9066428fa768f7c2925a4953"
I20260620 12:27:31.080009 13943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:31.081234 13943 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d9fd04006949d5998a43e60bd5c1bd in term 1.
I20260620 12:27:31.081583 13867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92d9fd04006949d5998a43e60bd5c1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93deb4c409f94f20a0e2d211bf153027"
I20260620 12:27:31.081722 13867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:31.082952 13867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d9fd04006949d5998a43e60bd5c1bd in term 1.
I20260620 12:27:31.083281 13768 leader_election.cc:302] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92d9fd04006949d5998a43e60bd5c1bd, 93deb4c409f94f20a0e2d211bf153027; no voters:
I20260620 12:27:31.083562 13978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:31.083822 13978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [term 1 LEADER]: Becoming Leader. State: Replica: 92d9fd04006949d5998a43e60bd5c1bd, State: Running, Role: LEADER
I20260620 12:27:31.084177 13978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } }
I20260620 12:27:31.089087 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d9fd04006949d5998a43e60bd5c1bd. Latest consensus state: current_term: 1 leader_uuid: "92d9fd04006949d5998a43e60bd5c1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:31.089203 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:31.089599 14027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:31.090669 14027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:31.096848 13943 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [term 1 FOLLOWER]: Refusing update from remote peer 92d9fd04006949d5998a43e60bd5c1bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:31.097440 14025 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:31.101429 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d9fd04006949d5998a43e60bd5c1bd. Latest consensus state: current_term: 1 leader_uuid: "92d9fd04006949d5998a43e60bd5c1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:31.101564 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:27:31.102432 13867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [term 1 FOLLOWER]: Refusing update from remote peer 92d9fd04006949d5998a43e60bd5c1bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:31.103001 13978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:31.106599 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d9fd04006949d5998a43e60bd5c1bd. Latest consensus state: current_term: 1 leader_uuid: "92d9fd04006949d5998a43e60bd5c1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:31.106756 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:27:31.114811 14025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92d9fd04006949d5998a43e60bd5c1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:31.114939 14025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:31.115994 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92d9fd04006949d5998a43e60bd5c1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:31.116106 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:27:31.116619 14028 mvcc.cc:205] Tried to move back new op lower bound from 7298901815686242304 to 7298901815649472512. Current Snapshot: MvccSnapshot[applied={T|T < 7298901815686242304}]
I20260620 12:27:31.117003 14031 mvcc.cc:205] Tried to move back new op lower bound from 7298901815686242304 to 7298901815649472512. Current Snapshot: MvccSnapshot[applied={T|T < 7298901815686242304}]
I20260620 12:27:31.118091 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92d9fd04006949d5998a43e60bd5c1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:31.118199 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:31.119129 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92d9fd04006949d5998a43e60bd5c1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d9fd04006949d5998a43e60bd5c1bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40639 } } peers { permanent_uuid: "93deb4c409f94f20a0e2d211bf153027" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43845 } } peers { permanent_uuid: "cc95ff5c9066428fa768f7c2925a4953" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37507 } } }
I20260620 12:27:31.119247 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:27:31.121608 14029 mvcc.cc:205] Tried to move back new op lower bound from 7298901815686242304 to 7298901815649472512. Current Snapshot: MvccSnapshot[applied={T|T < 7298901815686242304}]
I20260620 12:27:31.126278 14027 catalog_manager.cc:1347] Generated new cluster ID: a664362a62774f55a1c242cec202ac92
I20260620 12:27:31.126339 14027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:27:31.137251 14027 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:27:31.138464 14027 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:27:31.150118 14027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92d9fd04006949d5998a43e60bd5c1bd: Generated new TSK 0
I20260620 12:27:31.150925 14027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:27:31.943150 14008 catalog_manager.cc:1259] Loaded cluster ID: a664362a62774f55a1c242cec202ac92
I20260620 12:27:31.943231 14008 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: loading cluster ID for follower catalog manager: success
I20260620 12:27:31.944839 14008 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: acquiring CA information for follower catalog manager: success
I20260620 12:27:31.946048 14008 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93deb4c409f94f20a0e2d211bf153027: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:27:31.960444 14023 catalog_manager.cc:1259] Loaded cluster ID: a664362a62774f55a1c242cec202ac92
I20260620 12:27:31.960523 14023 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: loading cluster ID for follower catalog manager: success
I20260620 12:27:31.962074 14023 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: acquiring CA information for follower catalog manager: success
I20260620 12:27:31.963210 14023 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc95ff5c9066428fa768f7c2925a4953: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:27:32.217623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13750
I20260620 12:27:32.223906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13825
I20260620 12:27:32.229532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 13900
2026-06-20T12:27:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3140 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:27:32 dist-test-slave-bwpp krb5kdc[14041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:27:32 dist-test-slave-bwpp krb5kdc[14041](info): set up 2 sockets
Jun 20 12:27:32 dist-test-slave-bwpp krb5kdc[14041](info): commencing operation
krb5kdc: starting...
W20260620 12:27:33.279101 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:27:33 dist-test-slave-bwpp krb5kdc[14041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:27:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:27:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:27:33.397356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:33.505393 14054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:33.505632 14054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:33.505671 14054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:33.509593 14054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:33.509644 14054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:33.509663 14054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:33.509720 14054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:33.509749 14054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:33.514964 14054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:33.515914 14054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:33.516891 14054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:33.525337 14062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:33.525393 14060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:33.525394 14059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:33.525717 14054 server_base.cc:1034] running on GCE node
I20260620 12:27:33.526273 14054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:33.527341 14054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:33.528523 14054 hybrid_clock.cc:648] HybridClock initialized: now 1781958453528491 us; error 47 us; skew 500 ppm
Jun 20 12:27:33 dist-test-slave-bwpp krb5kdc[14041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:33.531174 14054 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:33.532051 14054 webserver.cc:460] Webserver started at http://127.0.98.62:44563/ using document root <none> and password file <none>
I20260620 12:27:33.532586 14054 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:33.532675 14054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:33.532905 14054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:33.534869 14054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7"
format_stamp: "Formatted at 2026-06-20 12:27:33 on dist-test-slave-bwpp"
server_key: "2c4da957d917b86a9065e67212e3af53"
server_key_iv: "4a08f1dfef1dd908bd990984473c0da8"
server_key_version: "encryptionkey@0"
I20260620 12:27:33.535408 14054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7"
format_stamp: "Formatted at 2026-06-20 12:27:33 on dist-test-slave-bwpp"
server_key: "2c4da957d917b86a9065e67212e3af53"
server_key_iv: "4a08f1dfef1dd908bd990984473c0da8"
server_key_version: "encryptionkey@0"
I20260620 12:27:33.538839 14054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:27:33.540970 14069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:33.542063 14054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:33.542196 14054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7"
format_stamp: "Formatted at 2026-06-20 12:27:33 on dist-test-slave-bwpp"
server_key: "2c4da957d917b86a9065e67212e3af53"
server_key_iv: "4a08f1dfef1dd908bd990984473c0da8"
server_key_version: "encryptionkey@0"
I20260620 12:27:33.542302 14054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:33.550907 14054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:33.553401 14054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:33.553584 14054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:33.560079 14054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37067
I20260620 12:27:33.560111 14121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37067 every 8 connection(s)
I20260620 12:27:33.560936 14054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:33.562508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14054
I20260620 12:27:33.562634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:27:33.562961 392 external_mini_cluster.cc:1428] Setting key 0667837df33d9240ba4fcc5838c98579
I20260620 12:27:33.563798 14122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:33.569077 14122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7: Bootstrap starting.
Jun 20 12:27:33 dist-test-slave-bwpp krb5kdc[14041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:33.571372 14122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:33.572224 14122 log.cc:824] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:33.574349 14122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7: No bootstrap required, opened a new log
I20260620 12:27:33.576720 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:33.564269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.0.98.62:37067)
0620 12:27:33.564667 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:33.564681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:33.564714 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:27:33.565303 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:33.565320 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:33.565612 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:33.565931 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:33.566759 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:33.567711 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:33.568567 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:33.569025 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:33.569192 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:33.571818 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:33.571823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:33.571835 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:33.571860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:33.573525 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:33.574029 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:33.574038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:33.574045 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:33.574203 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:33.574524 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:33.574527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:33.574529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:33.575047 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:33.575243 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:33.575461 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 12:27:33.575739 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260620 12:27:33.578871 14122 raft_consensus.cc:348] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37067 } }
I20260620 12:27:33.579177 14122 raft_consensus.cc:374] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:33.579290 14122 raft_consensus.cc:729] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76e9fd8ce7b4a3694970ac0cfb26aa7, State: Initialized, Role: FOLLOWER
I20260620 12:27:33.579641 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:27:33.579783 14122 consensus_queue.cc:260] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37067 } }
I20260620 12:27:33.579945 14122 raft_consensus.cc:388] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:27:33.580011 14122 raft_consensus.cc:482] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:27:33.580114 14122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:33.581336 14122 raft_consensus.cc:504] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37067 } }
I20260620 12:27:33.581637 14122 leader_election.cc:302] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b76e9fd8ce7b4a3694970ac0cfb26aa7; no voters:
I20260620 12:27:33.581887 14122 leader_election.cc:288] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:27:33.582000 14127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:33.582223 14127 raft_consensus.cc:686] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [term 1 LEADER]: Becoming Leader. State: Replica: b76e9fd8ce7b4a3694970ac0cfb26aa7, State: Running, Role: LEADER
I20260620 12:27:33.582553 14127 consensus_queue.cc:237] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37067 } }
I20260620 12:27:33.583124 14122 sys_catalog.cc:564] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:33.584183 14128 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37067 } } }
I20260620 12:27:33.584319 14128 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:33.584704 14136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:33.584678 14129 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b76e9fd8ce7b4a3694970ac0cfb26aa7. Latest consensus state: current_term: 1 leader_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76e9fd8ce7b4a3694970ac0cfb26aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37067 } } }
I20260620 12:27:33.585311 14129 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:33.587747 14136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:33.592726 14136 catalog_manager.cc:1347] Generated new cluster ID: 7d8bff1ef75647debbe01b8c83bff4b7
I20260620 12:27:33.592823 14136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:27:33.605201 14136 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:27:33.606104 14136 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:27:33.611446 14136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b76e9fd8ce7b4a3694970ac0cfb26aa7: Generated new TSK 0
I20260620 12:27:33.612053 14136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:27:33.635900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14054
2026-06-20T12:27:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1404 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:27:33 dist-test-slave-bwpp krb5kdc[14149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:27:33 dist-test-slave-bwpp krb5kdc[14149](info): set up 2 sockets
Jun 20 12:27:33 dist-test-slave-bwpp krb5kdc[14149](info): commencing operation
krb5kdc: starting...
W20260620 12:27:34.674957 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:27:34 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:27:34.794813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:34.901347 14162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:34.901598 14162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:34.901636 14162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:34.905596 14162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:34.905649 14162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:34.905668 14162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:34.905730 14162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:34.905761 14162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:34.910872 14162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:34.911893 14162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:34.913307 14162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:34.921701 14168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:34.921701 14167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:34.921705 14170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:34.921878 14162 server_base.cc:1034] running on GCE node
I20260620 12:27:34.922545 14162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:34.923707 14162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:34.924898 14162 hybrid_clock.cc:648] HybridClock initialized: now 1781958454924866 us; error 40 us; skew 500 ppm
Jun 20 12:27:34 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:34.932045 14162 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:34.933019 14162 webserver.cc:460] Webserver started at http://127.0.98.62:46425/ using document root <none> and password file <none>
I20260620 12:27:34.933578 14162 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:34.933637 14162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:34.933887 14162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:34.936003 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "ad3e27a79bc54adfbf09ab08f65312f0"
format_stamp: "Formatted at 2026-06-20 12:27:34 on dist-test-slave-bwpp"
server_key: "e09ee1d88f17afea5d23625a6dd80170"
server_key_iv: "066e27ecd04756ab96750b6f7c254818"
server_key_version: "encryptionkey@0"
I20260620 12:27:34.936599 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad3e27a79bc54adfbf09ab08f65312f0"
format_stamp: "Formatted at 2026-06-20 12:27:34 on dist-test-slave-bwpp"
server_key: "e09ee1d88f17afea5d23625a6dd80170"
server_key_iv: "066e27ecd04756ab96750b6f7c254818"
server_key_version: "encryptionkey@0"
I20260620 12:27:34.940404 14162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:27:34.943033 14177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:34.944380 14162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:27:34.944530 14162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "ad3e27a79bc54adfbf09ab08f65312f0"
format_stamp: "Formatted at 2026-06-20 12:27:34 on dist-test-slave-bwpp"
server_key: "e09ee1d88f17afea5d23625a6dd80170"
server_key_iv: "066e27ecd04756ab96750b6f7c254818"
server_key_version: "encryptionkey@0"
I20260620 12:27:34.944643 14162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:34.971181 14162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:34.974189 14162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:34.974408 14162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:34.982236 14162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40239
I20260620 12:27:34.982538 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40239 every 8 connection(s)
I20260620 12:27:34.983284 14162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:34.986025 14230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:34.991384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14162
I20260620 12:27:34.991540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:27:34.991616 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Bootstrap starting.
I20260620 12:27:34.991853 392 external_mini_cluster.cc:1428] Setting key cab4cbf2a53d85c07709487047f22b5a
I20260620 12:27:34.994199 14230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:34.995024 14230 log.cc:824] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:34.996937 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: No bootstrap required, opened a new log
Jun 20 12:27:34 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:35.001269 14230 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:35.001513 14230 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:35.001590 14230 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3e27a79bc54adfbf09ab08f65312f0, State: Initialized, Role: FOLLOWER
I20260620 12:27:35.002090 14230 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:35.002281 14230 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:27:35.002355 14230 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:27:35.002456 14230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:35.003371 14230 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:35.003715 14230 leader_election.cc:302] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad3e27a79bc54adfbf09ab08f65312f0; no voters:
I20260620 12:27:35.004055 14230 leader_election.cc:288] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:27:35.004117 14235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:35.004290 14235 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 LEADER]: Becoming Leader. State: Replica: ad3e27a79bc54adfbf09ab08f65312f0, State: Running, Role: LEADER
I20260620 12:27:35.004649 14235 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:35.005303 14230 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:35.007040 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:34.993188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.0.98.62:40239)
0620 12:27:34.993586 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:34.993612 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:34.993648 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:27:34.994321 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:34.994351 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:34.994724 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:34.995006 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:34.995875 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:34.996887 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:34.998032 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:34.998493 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:34.998609 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.001472 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:35.001476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:35.001489 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:35.001537 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:35.003448 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.003887 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.003893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.003899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.004061 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.004421 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.004426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.004428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.004932 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:35.005136 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.005770 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.006010 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
I20260620 12:27:35.007212 14236 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3e27a79bc54adfbf09ab08f65312f0. Latest consensus state: current_term: 1 leader_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260620 12:27:35.007325 14236 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:35.007779 14237 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260620 12:27:35.008077 14237 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: This master's current role is: LEADER
W20260620 12:27:35.010306 14251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:27:35.010394 14251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:27:35.010470 14250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:35.011253 14250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:35.016230 14250 catalog_manager.cc:1347] Generated new cluster ID: 159e4b997c794c86971b1b45c40bccde
I20260620 12:27:35.016327 14250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:27:35.034224 14250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:27:35.035190 14250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:27:35.041003 14250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Generated new TSK 0
I20260620 12:27:35.041808 14250 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:27:35.073725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:35.187511 14256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:35.187798 14256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:35.187847 14256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:35.192021 14256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:35.192083 14256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:35.192179 14256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:27:35.198099 14256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:35.199368 14256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:35.200428 14256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:35.210767 14262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:35.210767 14264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:35.210783 14261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:35.211262 14256 server_base.cc:1034] running on GCE node
I20260620 12:27:35.211761 14256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:35.212554 14256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:35.213759 14256 hybrid_clock.cc:648] HybridClock initialized: now 1781958455213748 us; error 43 us; skew 500 ppm
Jun 20 12:27:35 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:35.216987 14256 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:27:35.217952 14256 webserver.cc:460] Webserver started at http://127.0.98.1:44893/ using document root <none> and password file <none>
I20260620 12:27:35.218523 14256 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:35.218585 14256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:35.218834 14256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:35.220952 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d22252d3e9d4ca3a200d4872cd34ecf"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "7f4a6e5eb963094902276e50b7fe411e"
server_key_iv: "b1e7f76817981fc39eef642486b403fb"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.221514 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d22252d3e9d4ca3a200d4872cd34ecf"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "7f4a6e5eb963094902276e50b7fe411e"
server_key_iv: "b1e7f76817981fc39eef642486b403fb"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.225767 14256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 12:27:35.228513 14271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.229882 14256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:27:35.230053 14256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "0d22252d3e9d4ca3a200d4872cd34ecf"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "7f4a6e5eb963094902276e50b7fe411e"
server_key_iv: "b1e7f76817981fc39eef642486b403fb"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.230178 14256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:35.253537 14256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:35.257045 14256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:35.257259 14256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:35.257887 14256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:35.258803 14256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:35.258852 14256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.258926 14256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:35.258960 14256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.267701 14256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41681
I20260620 12:27:35.267719 14384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41681 every 8 connection(s)
I20260620 12:27:35.268620 14256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:27:35.270078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14256
I20260620 12:27:35.270226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:27:35.270620 392 external_mini_cluster.cc:1428] Setting key 5560447493492363280d447a9dd46b34
Jun 20 12:27:35 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:27:35.284070 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.270548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38541 (local address 127.0.98.62:40239)
0620 12:27:35.270724 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.270728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.272457 (+ 1729us) server_negotiation.cc:407] Connection header received
0620 12:27:35.273302 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.273308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.273372 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.273468 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:35.275135 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.275655 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.276856 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.277097 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.277207 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.280204 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:35.280211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:35.280214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:35.280249 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:35.281918 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.282417 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.282420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.282421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.282498 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.282877 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.282880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.282881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.283063 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:35.283138 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.283672 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.283848 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:27:35.284818 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.271028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:38541)
0620 12:27:35.272317 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.272359 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.273102 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.273696 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.273708 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.274262 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:35.275001 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.275017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.275789 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.275795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.276711 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.276721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.277248 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.277253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.277504 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.278236 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:35.278264 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:35.280028 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:35.282025 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:35.282030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:35.282041 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:35.282285 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:35.282584 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:35.282587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:35.282588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:35.282779 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:35.283152 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:35.283156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:35.283442 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.283601 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.283901 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":167,"threads_started":1}
I20260620 12:27:35.286180 14385 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260620 12:27:35.286456 14385 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:35.287005 14385 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260620 12:27:35.288524 14194 ts_manager.cc:194] Registered new tserver with Master: 0d22252d3e9d4ca3a200d4872cd34ecf (127.0.98.1:41681)
I20260620 12:27:35.289651 14194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:27:35.299674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:35.408306 14390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:35.408613 14390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:35.408681 14390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:35.412758 14390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:35.412858 14390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:35.412961 14390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:27:35.418224 14390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:35.419305 14390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:35.420337 14390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:35.429972 14398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:35.429946 14396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:35.429942 14395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:35.430007 14390 server_base.cc:1034] running on GCE node
I20260620 12:27:35.430763 14390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:35.431387 14390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:35.432547 14390 hybrid_clock.cc:648] HybridClock initialized: now 1781958455432531 us; error 26 us; skew 500 ppm
Jun 20 12:27:35 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:35.435384 14390 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:27:35.436216 14390 webserver.cc:460] Webserver started at http://127.0.98.2:40359/ using document root <none> and password file <none>
I20260620 12:27:35.436728 14390 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:35.436785 14390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:35.436947 14390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:35.439014 14390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "108aadfb220241f3aa20b9c8de306c7a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "91a96c7fcdb6b04b68659c17e71c1c22"
server_key_iv: "ff183dcccba93262531d820ec3947087"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.439534 14390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "108aadfb220241f3aa20b9c8de306c7a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "91a96c7fcdb6b04b68659c17e71c1c22"
server_key_iv: "ff183dcccba93262531d820ec3947087"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.443244 14390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:27:35.445611 14405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.446956 14390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:27:35.447101 14390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "108aadfb220241f3aa20b9c8de306c7a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "91a96c7fcdb6b04b68659c17e71c1c22"
server_key_iv: "ff183dcccba93262531d820ec3947087"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.447216 14390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:35.459057 14390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:35.465651 14390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:35.465857 14390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:35.466447 14390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:35.467326 14390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:35.467399 14390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.467470 14390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:35.467522 14390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.476442 14390 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32897
I20260620 12:27:35.476464 14518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32897 every 8 connection(s)
I20260620 12:27:35.477402 14390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:27:35.485879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14390
I20260620 12:27:35.485977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:27:35.486263 392 external_mini_cluster.cc:1428] Setting key bb834655e79c9a61424fb63dcd363608
Jun 20 12:27:35 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:35.491427 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.479334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46795 (local address 127.0.98.62:40239)
0620 12:27:35.479502 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.479506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.480529 (+ 1023us) server_negotiation.cc:407] Connection header received
0620 12:27:35.481368 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.481371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.481416 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.481493 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:35.482686 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.483234 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.484393 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.484625 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.484714 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.487425 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:35.487428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:35.487430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:35.487454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:35.489194 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.489719 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.489724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.489727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.489836 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.490246 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.490248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.490249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.490418 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:35.490493 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.491184 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.491289 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:27:35.492244 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.479735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:46795)
0620 12:27:35.480376 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.480415 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.481155 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.481654 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.481661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.482020 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:35.482487 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.482499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.483383 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.483386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.484254 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.484262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.484758 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.484760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.484898 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.485520 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:35.485536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:35.487261 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:35.489311 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:35.489316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:35.489327 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:35.489583 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:35.489937 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:35.489942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:35.489945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:35.490149 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:35.490529 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:35.490539 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:35.490908 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.491117 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.491371 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":111,"threads_started":1}
I20260620 12:27:35.493388 14519 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260620 12:27:35.493628 14519 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:35.494117 14519 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260620 12:27:35.495226 14194 ts_manager.cc:194] Registered new tserver with Master: 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897)
I20260620 12:27:35.496057 14194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46795
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:27:35.515780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:35.626019 14524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:35.626276 14524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:35.626369 14524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:35.630452 14524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:35.630505 14524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:35.630614 14524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:27:35.635840 14524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:35.636960 14524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:35.637966 14524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:35.647177 14532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:35.647174 14530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:35.647203 14529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:35.647435 14524 server_base.cc:1034] running on GCE node
I20260620 12:27:35.647933 14524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:35.648507 14524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:35.649690 14524 hybrid_clock.cc:648] HybridClock initialized: now 1781958455649673 us; error 34 us; skew 500 ppm
Jun 20 12:27:35 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:35.652654 14524 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:27:35.653470 14524 webserver.cc:460] Webserver started at http://127.0.98.3:46545/ using document root <none> and password file <none>
I20260620 12:27:35.654068 14524 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:35.654157 14524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:35.654385 14524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:35.656414 14524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f73646260804422a12a4e42dcb1ee6a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "03b82957c7c9e32e973fc4ad747b572d"
server_key_iv: "492449483d00e222bdc14ff142e2d77e"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.656992 14524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f73646260804422a12a4e42dcb1ee6a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "03b82957c7c9e32e973fc4ad747b572d"
server_key_iv: "492449483d00e222bdc14ff142e2d77e"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.660570 14524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:27:35.663046 14539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.664182 14524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:27:35.664326 14524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "8f73646260804422a12a4e42dcb1ee6a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "03b82957c7c9e32e973fc4ad747b572d"
server_key_iv: "492449483d00e222bdc14ff142e2d77e"
server_key_version: "encryptionkey@0"
I20260620 12:27:35.664438 14524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:35.693307 14524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:35.696580 14524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:35.696789 14524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:35.697381 14524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:35.698199 14524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:35.698271 14524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.698343 14524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:35.698386 14524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:35.707309 14524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40573
I20260620 12:27:35.707360 14652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40573 every 8 connection(s)
I20260620 12:27:35.708213 14524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:27:35.712090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14524
I20260620 12:27:35.712214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:27:35.712478 392 external_mini_cluster.cc:1428] Setting key 2992037dede3c904bd15ee875e517d07
Jun 20 12:27:35 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:35.721575 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.710073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45409 (local address 127.0.98.62:40239)
0620 12:27:35.710217 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.710220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.711301 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 12:27:35.712204 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.712207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.712268 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.712357 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:35.713785 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.714396 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.715588 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.715803 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.715885 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.718083 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:35.718086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:35.718087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:35.718112 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:35.719606 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.720089 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.720094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.720096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.720178 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.720509 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.720512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.720514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.720690 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:35.720801 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.721282 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.721398 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:27:35.722319 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.710476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:45409)
0620 12:27:35.711149 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.711188 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.712000 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.712567 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.712578 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.713067 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:35.713634 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.713654 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.714520 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.714524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.715482 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.715491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.715938 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.715941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.716073 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.716576 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:35.716591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:35.717946 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:35.719721 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:35.719727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:35.719742 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:35.719987 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:35.720277 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:35.720285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:35.720286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:35.720405 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:35.720796 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:35.720802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:35.721035 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.721229 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.721522 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":138,"threads_started":1}
I20260620 12:27:35.723482 14653 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260620 12:27:35.723701 14653 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:35.724145 14653 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260620 12:27:35.725126 14194 ts_manager.cc:194] Registered new tserver with Master: 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
I20260620 12:27:35.725845 14194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45409
I20260620 12:27:35.726727 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:27:35.740628 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.730093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.0.98.62:40239)
0620 12:27:35.730258 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.730262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.730590 (+ 328us) server_negotiation.cc:407] Connection header received
0620 12:27:35.730826 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.730832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.730880 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.730988 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:35.732029 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.732748 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.734228 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.734546 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.734675 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.736676 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:35.736680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:35.736681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:35.736702 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:35.738426 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.739184 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.739189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.739191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.739279 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:35.739570 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:35.739575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:35.739577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:35.739813 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:35.739953 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.740291 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.740463 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 12:27:35.745242 14193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:27:35.749274 14193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:27:35.766306 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.760998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32897 (local address 127.0.0.1:53584)
0620 12:27:35.761198 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.761211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.761294 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.762168 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.762171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.762188 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:35.762272 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.762277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.763760 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.763764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.764742 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.764752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.765756 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.765759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.765864 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.765905 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.765988 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.766107 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260620 12:27:35.766351 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.760361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41681 (local address 127.0.0.1:37242)
0620 12:27:35.760583 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.760596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.760679 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.761405 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.761408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.761428 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:35.761529 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.761536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.763042 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.763045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.764407 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.764418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.765890 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.765894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.766026 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.766049 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.766112 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.766178 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":134,"thread_start_us":90,"threads_started":1}
I20260620 12:27:35.766834 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.759921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.0.98.3:40573)
0620 12:27:35.760270 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.760279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.760299 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:27:35.760398 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.760411 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.760619 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.761138 (+ 519us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:35.761291 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.762731 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.764929 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.765747 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.766048 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.766205 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.766560 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.766698 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":152,"threads_started":1}
I20260620 12:27:35.766729 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.759168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40573 (local address 127.0.0.1:52980)
0620 12:27:35.759461 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.759516 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.759633 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.760818 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.760824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.760848 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:35.760987 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.760995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.762871 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.762875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.764828 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.764835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.765890 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.765894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.766413 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.766444 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.766526 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.766593 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":879,"thread_start_us":80,"threads_started":1}
I20260620 12:27:35.767268 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.760551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.0.98.1:41681)
0620 12:27:35.760936 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.760952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.760975 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:27:35.761050 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.761057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.761252 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.761677 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:35.761821 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.762898 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.764973 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.765747 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.766797 (+ 1050us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.766978 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.767029 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.767115 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":88,"threads_started":1}
I20260620 12:27:35.767413 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.761400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.0.98.2:32897)
0620 12:27:35.761696 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.761713 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.761735 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:27:35.761810 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.761814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.762010 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.762401 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:35.762540 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.763608 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.764892 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.765627 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.766990 (+ 1363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.767133 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.767182 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.767278 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":128,"threads_started":1}
I20260620 12:27:35.774515 14587 tablet_service.cc:1467] Processing CreateTablet for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 (DEFAULT_TABLE table=test-table [id=9024566c038240ce8e741be4119fa1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:35.775262 14453 tablet_service.cc:1467] Processing CreateTablet for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 (DEFAULT_TABLE table=test-table [id=9024566c038240ce8e741be4119fa1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:35.775260 14319 tablet_service.cc:1467] Processing CreateTablet for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 (DEFAULT_TABLE table=test-table [id=9024566c038240ce8e741be4119fa1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:35.775803 14587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:35.776264 14453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:35.776264 14319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:35.783578 14670 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap starting.
I20260620 12:27:35.785135 14672 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Bootstrap starting.
I20260620 12:27:35.785943 14670 tablet_bootstrap.cc:654] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:35.787285 14670 log.cc:824] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:35.787711 14671 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Bootstrap starting.
I20260620 12:27:35.788434 14672 tablet_bootstrap.cc:654] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:35.789413 14670 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: No bootstrap required, opened a new log
I20260620 12:27:35.789600 14670 ts_tablet_manager.cc:1397] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 12:27:35.790513 14671 tablet_bootstrap.cc:654] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:35.791725 14671 log.cc:824] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:35.792881 14672 log.cc:824] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:35.793556 14670 raft_consensus.cc:348] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.793797 14671 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: No bootstrap required, opened a new log
I20260620 12:27:35.793839 14670 raft_consensus.cc:374] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:35.793937 14670 raft_consensus.cc:729] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d22252d3e9d4ca3a200d4872cd34ecf, State: Initialized, Role: FOLLOWER
I20260620 12:27:35.793998 14671 ts_tablet_manager.cc:1397] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:27:35.794392 14670 consensus_queue.cc:260] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.794852 14672 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: No bootstrap required, opened a new log
I20260620 12:27:35.795008 14385 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260620 12:27:35.795065 14672 ts_tablet_manager.cc:1397] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 12:27:35.795439 14670 ts_tablet_manager.cc:1428] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 12:27:35.798472 14672 raft_consensus.cc:348] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.798786 14672 raft_consensus.cc:374] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:35.798851 14672 raft_consensus.cc:729] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108aadfb220241f3aa20b9c8de306c7a, State: Initialized, Role: FOLLOWER
I20260620 12:27:35.798707 14671 raft_consensus.cc:348] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.798965 14671 raft_consensus.cc:374] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:35.799046 14671 raft_consensus.cc:729] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73646260804422a12a4e42dcb1ee6a, State: Initialized, Role: FOLLOWER
I20260620 12:27:35.799312 14672 consensus_queue.cc:260] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.799507 14671 consensus_queue.cc:260] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.799954 14519 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260620 12:27:35.800211 14653 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260620 12:27:35.800246 14672 ts_tablet_manager.cc:1428] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:27:35.800488 14671 ts_tablet_manager.cc:1428] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:27:35.857602 14676 raft_consensus.cc:482] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:35.857846 14676 raft_consensus.cc:504] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.859174 14676 leader_election.cc:288] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573), 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897)
I20260620 12:27:35.864804 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.859524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59409 (local address 127.0.98.3:40573)
0620 12:27:35.859689 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.859695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.859997 (+ 302us) server_negotiation.cc:407] Connection header received
0620 12:27:35.860161 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.860164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.860220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.860307 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:35.860736 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.861525 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.863482 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.864122 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.864223 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.864287 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.864594 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.864671 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 12:27:35.864804 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.859560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52263 (local address 127.0.98.2:32897)
0620 12:27:35.859674 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.859679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.860072 (+ 393us) server_negotiation.cc:407] Connection header received
0620 12:27:35.860160 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.860164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.860220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.860308 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:35.860739 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.861544 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.863479 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.864112 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.864223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.864287 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.864594 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.864671 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 12:27:35.864866 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.859430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32897 (local address 127.0.98.1:52263)
0620 12:27:35.859986 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.859998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.860081 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.860334 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.860337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.860378 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:35.860613 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.860621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.861649 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.861652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.863343 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.863355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.864256 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.864260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.864416 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.864466 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.864545 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.864677 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":370,"spinlock_wait_cycles":29184,"thread_start_us":108,"threads_started":1}
I20260620 12:27:35.864868 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.859428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40573 (local address 127.0.98.1:59409)
0620 12:27:35.859887 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.859902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:35.860035 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:35.860334 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:35.860337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:35.860380 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:35.860613 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.860621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.861649 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.861653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.863343 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:35.863355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.864256 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.864260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:35.864420 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.864461 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 12:27:35.864545 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 12:27:35.864679 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":301,"spinlock_wait_cycles":6656,"thread_start_us":137,"threads_started":1}
I20260620 12:27:35.865556 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b91080ec4ad4ef0971cf4eba9ac15c3" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108aadfb220241f3aa20b9c8de306c7a" is_pre_election: true
I20260620 12:27:35.865813 14473 raft_consensus.cc:2454] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 0.
I20260620 12:27:35.866308 14607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b91080ec4ad4ef0971cf4eba9ac15c3" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73646260804422a12a4e42dcb1ee6a" is_pre_election: true
I20260620 12:27:35.866321 14275 leader_election.cc:302] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf, 108aadfb220241f3aa20b9c8de306c7a; no voters:
I20260620 12:27:35.866598 14607 raft_consensus.cc:2454] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 0.
I20260620 12:27:35.866626 14676 raft_consensus.cc:2781] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:27:35.866778 14676 raft_consensus.cc:482] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:27:35.866837 14676 raft_consensus.cc:3037] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:35.867978 14676 raft_consensus.cc:504] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.868348 14676 leader_election.cc:288] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 election: Requested vote from peers 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573), 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897)
I20260620 12:27:35.868696 14607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b91080ec4ad4ef0971cf4eba9ac15c3" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73646260804422a12a4e42dcb1ee6a"
I20260620 12:27:35.868736 14473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b91080ec4ad4ef0971cf4eba9ac15c3" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108aadfb220241f3aa20b9c8de306c7a"
I20260620 12:27:35.868818 14607 raft_consensus.cc:3037] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:35.868924 14473 raft_consensus.cc:3037] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:35.869891 14607 raft_consensus.cc:2454] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 1.
I20260620 12:27:35.870007 14473 raft_consensus.cc:2454] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 1.
I20260620 12:27:35.870184 14272 leader_election.cc:302] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf, 8f73646260804422a12a4e42dcb1ee6a; no voters:
I20260620 12:27:35.870385 14676 raft_consensus.cc:2781] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:35.870640 14676 raft_consensus.cc:686] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 LEADER]: Becoming Leader. State: Replica: 0d22252d3e9d4ca3a200d4872cd34ecf, State: Running, Role: LEADER
I20260620 12:27:35.871023 14676 consensus_queue.cc:237] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:35.874083 14193 catalog_manager.cc:5495] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d22252d3e9d4ca3a200d4872cd34ecf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } health_report { overall_health: UNKNOWN } } }
I20260620 12:27:35.895563 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.891871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.98.1:41681)
0620 12:27:35.892015 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:35.892019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:35.892034 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:27:35.892199 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:35.892201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:35.892285 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:35.892407 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:27:35.892672 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.893185 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.894061 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:35.894282 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:35.894392 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:35.894728 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:27:35.895158 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:27:35.895298 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:35.895357 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 12:27:35.895413 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 12:27:35.901722 14607 raft_consensus.cc:1261] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Refusing update from remote peer 0d22252d3e9d4ca3a200d4872cd34ecf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:35.901721 14473 raft_consensus.cc:1261] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Refusing update from remote peer 0d22252d3e9d4ca3a200d4872cd34ecf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:35.902496 14681 consensus_queue.cc:1035] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:35.902685 14676 consensus_queue.cc:1035] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:35.910211 14684 mvcc.cc:205] Tried to move back new op lower bound from 7298901835365511168 to 7298901835250163712. Current Snapshot: MvccSnapshot[applied={T|T < 7298901835365511168}]
I20260620 12:27:35.912216 14685 mvcc.cc:205] Tried to move back new op lower bound from 7298901835365511168 to 7298901835250163712. Current Snapshot: MvccSnapshot[applied={T|T < 7298901835365511168}]
I20260620 12:27:35.918419 14193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:35.918586 14193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:35.920756 14193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Sending DeleteTablet for 3 replicas of tablet 7b91080ec4ad4ef0971cf4eba9ac15c3
I20260620 12:27:35.921664 14319 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37242
I20260620 12:27:35.921715 14587 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52980
I20260620 12:27:35.921701 14453 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53584
I20260620 12:27:35.921782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14256
I20260620 12:27:35.922693 14693 tablet_replica.cc:330] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: stopping tablet replica
I20260620 12:27:35.922847 14694 tablet_replica.cc:330] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: stopping tablet replica
I20260620 12:27:35.923028 14693 raft_consensus.cc:2229] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:35.923143 14694 raft_consensus.cc:2229] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:35.923341 14693 raft_consensus.cc:2258] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:35.923418 14694 raft_consensus.cc:2258] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:35.925071 14693 ts_tablet_manager.cc:1905] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:35.927758 14694 ts_tablet_manager.cc:1905] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:35.928834 14693 ts_tablet_manager.cc:1918] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:35.928948 14693 log.cc:1196] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/7b91080ec4ad4ef0971cf4eba9ac15c3
I20260620 12:27:35.929359 14693 ts_tablet_manager.cc:1939] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 8f73646260804422a12a4e42dcb1ee6a: Deleting consensus metadata
W20260620 12:27:35.929952 14179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41681 (error 108)
I20260620 12:27:35.930624 14694 ts_tablet_manager.cc:1918] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:35.930848 14694 log.cc:1196] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/7b91080ec4ad4ef0971cf4eba9ac15c3
I20260620 12:27:35.930824 14178 catalog_manager.cc:4841] TS 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573): tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 (table test-table [id=9024566c038240ce8e741be4119fa1ba]) successfully deleted
I20260620 12:27:35.931250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14390
I20260620 12:27:35.931236 14694 ts_tablet_manager.cc:1939] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 108aadfb220241f3aa20b9c8de306c7a: Deleting consensus metadata
I20260620 12:27:35.931226 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.930868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41681 (local address 127.0.0.1:37260)
0620 12:27:35.931019 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.931122 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 12:27:35.932844 14179 catalog_manager.cc:4584] TS 0d22252d3e9d4ca3a200d4872cd34ecf (127.0.98.1:41681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41681: connect: Connection refused (error 111)
I20260620 12:27:35.937489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14524
I20260620 12:27:35.939527 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:35.937690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32897 (local address 127.0.0.1:53586)
0620 12:27:35.939348 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:35.939419 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1540}
I20260620 12:27:35.943331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14162
I20260620 12:27:35.949734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--webserver_interface=127.0.98.62
--webserver_port=46425
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:36.059378 14696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:36.059669 14696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:36.059738 14696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:36.063694 14696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:36.063771 14696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:36.063812 14696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:36.063869 14696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:36.063905 14696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:36.069065 14696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:36.070106 14696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:36.071127 14696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:36.083851 14703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:36.083858 14702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:36.083833 14705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:36.084108 14696 server_base.cc:1034] running on GCE node
I20260620 12:27:36.084807 14696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:36.085988 14696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:36.087172 14696 hybrid_clock.cc:648] HybridClock initialized: now 1781958456087160 us; error 35 us; skew 500 ppm
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:36.089859 14696 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:36.090797 14696 webserver.cc:460] Webserver started at http://127.0.98.62:46425/ using document root <none> and password file <none>
I20260620 12:27:36.091324 14696 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:36.091390 14696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:36.095053 14696 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 12:27:36.097118 14712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.098280 14696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:36.098405 14696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "ad3e27a79bc54adfbf09ab08f65312f0"
format_stamp: "Formatted at 2026-06-20 12:27:34 on dist-test-slave-bwpp"
server_key: "e09ee1d88f17afea5d23625a6dd80170"
server_key_iv: "066e27ecd04756ab96750b6f7c254818"
server_key_version: "encryptionkey@0"
I20260620 12:27:36.099047 14696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:36.121088 14696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:36.123828 14696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:36.124029 14696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:36.131568 14696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40239
I20260620 12:27:36.131839 14764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40239 every 8 connection(s)
I20260620 12:27:36.132628 14696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:36.136353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14696
I20260620 12:27:36.136842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41681
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40239
--webserver_port=44893
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:36.137084 14765 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:27:36.138669 14765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Bootstrap starting.
I20260620 12:27:36.157424 14765 log.cc:824] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:36.165688 14765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:27:36.166064 14765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Bootstrap complete.
I20260620 12:27:36.170022 14765 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:36.170421 14765 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3e27a79bc54adfbf09ab08f65312f0, State: Initialized, Role: FOLLOWER
I20260620 12:27:36.170938 14765 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:36.171087 14765 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:27:36.171180 14765 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:27:36.171299 14765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:27:36.173061 14765 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:36.173385 14765 leader_election.cc:302] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad3e27a79bc54adfbf09ab08f65312f0; no voters:
I20260620 12:27:36.173704 14765 leader_election.cc:288] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:27:36.173785 14769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:27:36.173981 14769 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 LEADER]: Becoming Leader. State: Replica: ad3e27a79bc54adfbf09ab08f65312f0, State: Running, Role: LEADER
I20260620 12:27:36.174259 14769 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:36.174937 14765 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:36.175571 14770 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260620 12:27:36.175693 14770 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:36.175907 14770 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3e27a79bc54adfbf09ab08f65312f0. Latest consensus state: current_term: 2 leader_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260620 12:27:36.176005 14770 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:36.176667 14774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:36.181222 14774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:36.181825 14774 catalog_manager.cc:1259] Loaded cluster ID: 159e4b997c794c86971b1b45c40bccde
I20260620 12:27:36.181885 14774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:27:36.183303 14774 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 12:27:36.188133 14774 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 12:27:36.193478 14774 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:27:36.194175 14774 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Loaded TSK: 0
I20260620 12:27:36.195343 14774 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:27:36.252526 14767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:36.252851 14767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:36.252939 14767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:36.257925 14767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:36.258006 14767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:36.258101 14767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:27:36.263478 14767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44893
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:36.264516 14767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:36.265538 14767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:36.276134 14793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:36.276147 14795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:36.276480 14792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:36.276832 14767 server_base.cc:1034] running on GCE node
I20260620 12:27:36.277307 14767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:36.278053 14767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:36.279245 14767 hybrid_clock.cc:648] HybridClock initialized: now 1781958456279189 us; error 62 us; skew 500 ppm
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:36.282612 14767 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:27:36.283504 14767 webserver.cc:460] Webserver started at http://127.0.98.1:44893/ using document root <none> and password file <none>
I20260620 12:27:36.284060 14767 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:36.284168 14767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:36.288074 14767 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:27:36.290328 14802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.291714 14767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:27:36.291807 14767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "0d22252d3e9d4ca3a200d4872cd34ecf"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "7f4a6e5eb963094902276e50b7fe411e"
server_key_iv: "b1e7f76817981fc39eef642486b403fb"
server_key_version: "encryptionkey@0"
I20260620 12:27:36.292413 14767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:36.312522 14767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:36.315923 14767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:36.316139 14767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:36.316792 14767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:36.317966 14809 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:27:36.322075 14767 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:27:36.322137 14767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 12:27:36.322183 14767 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:27:36.323947 14767 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:27:36.324004 14767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 12:27:36.324254 14809 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap starting.
I20260620 12:27:36.334957 14767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41681
I20260620 12:27:36.335039 14916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41681 every 8 connection(s)
I20260620 12:27:36.336145 14767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:27:36.343339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14767
I20260620 12:27:36.343807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32897
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40239
--webserver_port=40359
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:36.344538 14809 log.cc:824] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:36.352665 14809 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:27:36.353101 14809 tablet_bootstrap.cc:492] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap complete.
I20260620 12:27:36.354206 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.338171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45045 (local address 127.0.98.62:40239)
0620 12:27:36.338524 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.338540 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.339095 (+ 555us) server_negotiation.cc:407] Connection header received
0620 12:27:36.340195 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.340206 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.340521 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.340852 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:36.341951 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.342971 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.344330 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.344849 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.345027 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.348123 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:36.348129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:36.348144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:36.348176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:36.350332 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.351059 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.351065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.351070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.351215 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.351642 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.351646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.351648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.352070 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:36.352202 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.352996 (+ 794us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.353350 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":134,"threads_started":1}
I20260620 12:27:36.354245 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.338199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:45045)
0620 12:27:36.338910 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:36.338951 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:36.339785 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:36.340898 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:36.340906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:36.341334 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:36.341769 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.341782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.343140 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.343143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.344200 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.344210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.345000 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.345003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.345188 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.345845 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:36.345870 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:36.347914 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:36.350465 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.350473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.350488 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.350919 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.351322 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.351325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.351328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.351522 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.352201 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:36.352209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:36.352539 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 12:27:36.352776 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 12:27:36.353148 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":141,"threads_started":1}
I20260620 12:27:36.354502 14809 ts_tablet_manager.cc:1397] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20260620 12:27:36.356933 14917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260620 12:27:36.357152 14917 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:36.356998 14809 raft_consensus.cc:348] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:36.357481 14809 raft_consensus.cc:729] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d22252d3e9d4ca3a200d4872cd34ecf, State: Initialized, Role: FOLLOWER
I20260620 12:27:36.357648 14917 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260620 12:27:36.357929 14809 consensus_queue.cc:260] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } } peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } }
I20260620 12:27:36.358924 14809 ts_tablet_manager.cc:1428] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent starting tablet: real 0.004s user 0.008s sys 0.000s
I20260620 12:27:36.360491 14729 ts_manager.cc:194] Registered new tserver with Master: 0d22252d3e9d4ca3a200d4872cd34ecf (127.0.98.1:41681)
I20260620 12:27:36.364440 14729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45045
I20260620 12:27:36.366561 14917 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:27:36.374044 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.363734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.0.98.1:41681)
0620 12:27:36.364388 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.364400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.364422 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:27:36.364486 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.364491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.364696 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.364840 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:36.365476 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.366626 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.367978 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.368362 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.368471 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.370820 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:36.370827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:36.370840 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:36.370861 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:36.372326 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.372824 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.372828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.372829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.372900 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.373159 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.373161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.373162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.373388 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:36.373483 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.373818 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.373935 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":79,"threads_started":1}
I20260620 12:27:36.374099 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.363692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41681 (local address 127.0.0.1:37274)
0620 12:27:36.364162 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:36.364202 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:36.364362 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:36.364840 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:36.364846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:36.365249 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:36.365362 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.365368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.366787 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.366790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.367777 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.367788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.368489 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.368504 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.368615 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.369125 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:36.369142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:36.370693 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:36.372432 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.372440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.372445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.372716 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.372982 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.372985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.372987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.373076 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.373491 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:36.373497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:36.373610 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 12:27:36.373752 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 12:27:36.373928 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":121,"threads_started":1}
I20260620 12:27:36.375267 14851 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37274
I20260620 12:27:36.376264 14926 tablet_replica.cc:330] stopping tablet replica
I20260620 12:27:36.376611 14926 raft_consensus.cc:2229] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:36.376854 14926 raft_consensus.cc:2258] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:36.377758 14926 ts_tablet_manager.cc:1905] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:36.381083 14926 ts_tablet_manager.cc:1918] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:36.381160 14926 log.cc:1196] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/7b91080ec4ad4ef0971cf4eba9ac15c3
I20260620 12:27:36.381428 14926 ts_tablet_manager.cc:1939] T 7b91080ec4ad4ef0971cf4eba9ac15c3 P 0d22252d3e9d4ca3a200d4872cd34ecf: Deleting consensus metadata
I20260620 12:27:36.382928 14714 catalog_manager.cc:4841] TS 0d22252d3e9d4ca3a200d4872cd34ecf (127.0.98.1:41681): tablet 7b91080ec4ad4ef0971cf4eba9ac15c3 (table test-table [id=9024566c038240ce8e741be4119fa1ba]) successfully deleted
W20260620 12:27:36.460044 14922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:36.460289 14922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:36.460363 14922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:36.465685 14922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:36.465739 14922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:36.465796 14922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:27:36.470790 14922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40359
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:36.471837 14922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:36.472770 14922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:36.482501 14932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:36.482704 14922 server_base.cc:1034] running on GCE node
W20260620 12:27:36.482540 14934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:36.482502 14931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:36.483240 14922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:36.483851 14922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:36.485034 14922 hybrid_clock.cc:648] HybridClock initialized: now 1781958456485021 us; error 24 us; skew 500 ppm
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:36.487558 14922 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:27:36.488397 14922 webserver.cc:460] Webserver started at http://127.0.98.2:40359/ using document root <none> and password file <none>
I20260620 12:27:36.488926 14922 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:36.489023 14922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:36.492625 14922 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:27:36.494500 14941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.495754 14922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:36.495885 14922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "108aadfb220241f3aa20b9c8de306c7a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "91a96c7fcdb6b04b68659c17e71c1c22"
server_key_iv: "ff183dcccba93262531d820ec3947087"
server_key_version: "encryptionkey@0"
I20260620 12:27:36.496462 14922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:36.509330 14922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:36.511889 14922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:36.512069 14922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:36.512570 14922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:36.513314 14922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:36.513392 14922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.513476 14922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:36.513526 14922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.522122 14922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32897
I20260620 12:27:36.522143 15054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32897 every 8 connection(s)
I20260620 12:27:36.523103 14922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:27:36.530893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 14922
I20260620 12:27:36.531371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40573
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40239
--webserver_port=46545
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:36.535815 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.524798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47979 (local address 127.0.98.62:40239)
0620 12:27:36.524930 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.524934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.525815 (+ 881us) server_negotiation.cc:407] Connection header received
0620 12:27:36.526680 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.526682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.526728 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.526794 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:36.527902 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.528414 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.529477 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.529691 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.529764 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.532401 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:36.532404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:36.532406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:36.532430 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:36.533812 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.534307 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.534312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.534313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.534391 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.534792 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.534795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.534797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.534987 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:36.535090 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.535535 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.535653 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:27:36.536506 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.525128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:47979)
0620 12:27:36.525707 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:36.525743 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:36.526488 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:36.526922 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:36.526950 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:36.527312 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:36.527788 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.527802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.528531 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.528541 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.529377 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.529386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.529808 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.529811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.529937 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.530518 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:36.530533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:36.532213 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:36.533946 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.533951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.533963 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.534205 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.534483 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.534487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.534489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.534620 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.535089 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:36.535095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:36.535315 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 12:27:36.535497 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 12:27:36.535776 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":102,"threads_started":1}
I20260620 12:27:36.537853 15055 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260620 12:27:36.538070 15055 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:36.538534 15055 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260620 12:27:36.539671 14729 ts_manager.cc:194] Registered new tserver with Master: 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897)
I20260620 12:27:36.540539 14729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47979
W20260620 12:27:36.641163 15058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:36.641410 15058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:36.641450 15058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:36.645355 15058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:36.645407 15058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:36.645462 15058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:27:36.650395 15058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46545
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:36.651389 15058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:36.652310 15058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:36.661882 15063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:36.661975 15058 server_base.cc:1034] running on GCE node
W20260620 12:27:36.661885 15066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:36.661955 15064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:36.662554 15058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:36.663264 15058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:36.664444 15058 hybrid_clock.cc:648] HybridClock initialized: now 1781958456664428 us; error 27 us; skew 500 ppm
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:36.667572 15058 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:27:36.668431 15058 webserver.cc:460] Webserver started at http://127.0.98.3:46545/ using document root <none> and password file <none>
I20260620 12:27:36.669003 15058 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:36.669102 15058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:36.673259 15058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 12:27:36.675390 15073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.676687 15058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:36.676811 15058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "8f73646260804422a12a4e42dcb1ee6a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "03b82957c7c9e32e973fc4ad747b572d"
server_key_iv: "492449483d00e222bdc14ff142e2d77e"
server_key_version: "encryptionkey@0"
I20260620 12:27:36.677450 15058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:36.697424 15058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:36.700728 15058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:36.700924 15058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:36.701464 15058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:36.702391 15058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:36.702489 15058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.702553 15058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:36.702600 15058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:36.711392 15058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40573
I20260620 12:27:36.711454 15186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40573 every 8 connection(s)
I20260620 12:27:36.712376 15058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:27:36.717406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 15058
Jun 20 12:27:36 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:36.726622 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.718129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.0.98.62:40239)
0620 12:27:36.718384 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.718389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.718504 (+ 115us) server_negotiation.cc:407] Connection header received
0620 12:27:36.718629 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.718631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.718720 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.718805 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:36.719683 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.720363 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.721522 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.721734 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.721820 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.723257 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:36.723261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:36.723263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:36.723307 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:36.724905 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.725313 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.725317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.725319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.725405 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.725701 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.725705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.725707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.725887 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:36.726108 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.726287 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.726402 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260620 12:27:36.727008 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.714150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40469 (local address 127.0.98.62:40239)
0620 12:27:36.714281 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.714284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.715424 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 12:27:36.716301 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.716304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.716361 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.716443 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:36.717614 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.718311 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.719997 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.720330 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.720444 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.723140 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:36.723145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:36.723152 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:36.723173 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:36.724792 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.725281 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.725285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.725289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.725361 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:36.725711 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:36.725714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:36.725715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:36.725907 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:36.726027 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.726489 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.726598 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:27:36.727715 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.714569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:40469)
0620 12:27:36.715281 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:36.715321 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:36.716101 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:36.716623 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:36.716630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:36.716957 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:36.717399 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.717411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.718481 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.718486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.719886 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.719896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.720510 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.720514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.720702 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.721318 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:36.721338 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:36.722970 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:36.724907 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.724912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.724926 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.725174 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.725463 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:36.725466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:36.725468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:36.725613 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:36.726027 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:36.726034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:36.726254 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 12:27:36.726433 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 12:27:36.726769 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":116,"threads_started":1}
I20260620 12:27:36.728891 15187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260620 12:27:36.729105 15187 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:36.729566 15187 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260620 12:27:36.730465 14729 ts_manager.cc:194] Registered new tserver with Master: 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
I20260620 12:27:36.731033 14729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40469
I20260620 12:27:36.740082 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:27:36.743894 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.740705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.0.98.62:40239)
0620 12:27:36.740823 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.740826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.741016 (+ 190us) server_negotiation.cc:407] Connection header received
0620 12:27:36.741105 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.741107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.741193 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.741317 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:27:36.741465 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.741922 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.742769 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.742977 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.743057 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.743306 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:27:36.743621 (+ 315us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:27:36.743706 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.743762 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.743812 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:27:36.744424 14729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:27:36.747583 14729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:27:36.754741 14851 tablet_service.cc:1467] Processing CreateTablet for tablet cc7b962fb9f54c138ccb276f27af7604 (DEFAULT_TABLE table=test-table [id=dda5ebd5df4848239a896f1757e35a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:36.755362 14851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7b962fb9f54c138ccb276f27af7604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:36.758190 14809 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap starting.
I20260620 12:27:36.758898 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.753616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32897 (local address 127.0.0.1:60398)
0620 12:27:36.753723 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:36.753736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:36.753803 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:36.754868 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:36.754870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:36.754886 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:36.754992 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.755000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.756236 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.756242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.757354 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.757362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.758404 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.758407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.758518 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.758562 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 12:27:36.758700 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 12:27:36.758777 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":24}
I20260620 12:27:36.758988 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.753826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.98.2:32897)
0620 12:27:36.754272 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.754284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.754353 (+ 69us) server_negotiation.cc:407] Connection header received
0620 12:27:36.754432 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.754437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.754677 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.755136 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:36.755321 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.756099 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.757467 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.758032 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.758155 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.758290 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.758748 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.758843 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":134,"threads_started":1}
I20260620 12:27:36.759093 14809 tablet_bootstrap.cc:654] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:36.759898 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.754040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40573 (local address 127.0.0.1:42786)
0620 12:27:36.754263 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:36.754277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:36.754360 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:36.755303 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:36.755307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:36.755324 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:36.755399 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.755404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.756639 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.756643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.757925 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:36.757935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.759539 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.759543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:36.759626 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.759648 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:27:36.759692 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 12:27:36.759756 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260620 12:27:36.760514 14809 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: No bootstrap required, opened a new log
I20260620 12:27:36.760604 14809 ts_tablet_manager.cc:1397] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:36.760710 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:36.754432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.98.3:40573)
0620 12:27:36.754790 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:36.754878 (+ 88us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:36.754903 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:27:36.754971 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:36.754975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:36.755169 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:36.755310 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:36.755503 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.756452 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.758087 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:36.759392 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:36.760274 (+ 882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:36.760421 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:36.760470 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 12:27:36.760532 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260620 12:27:36.761024 14809 raft_consensus.cc:348] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:36.761142 14809 raft_consensus.cc:374] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:36.761194 14809 raft_consensus.cc:729] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d22252d3e9d4ca3a200d4872cd34ecf, State: Initialized, Role: FOLLOWER
I20260620 12:27:36.761307 14809 consensus_queue.cc:260] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:36.761561 14809 ts_tablet_manager.cc:1428] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:27:36.762076 14989 tablet_service.cc:1467] Processing CreateTablet for tablet cc7b962fb9f54c138ccb276f27af7604 (DEFAULT_TABLE table=test-table [id=dda5ebd5df4848239a896f1757e35a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:36.763042 14989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7b962fb9f54c138ccb276f27af7604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:36.762867 15121 tablet_service.cc:1467] Processing CreateTablet for tablet cc7b962fb9f54c138ccb276f27af7604 (DEFAULT_TABLE table=test-table [id=dda5ebd5df4848239a896f1757e35a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:36.763960 15121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7b962fb9f54c138ccb276f27af7604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:36.769762 15200 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Bootstrap starting.
I20260620 12:27:36.770063 15201 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Bootstrap starting.
I20260620 12:27:36.772028 15201 tablet_bootstrap.cc:654] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:36.772207 15200 tablet_bootstrap.cc:654] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:36.773165 15200 log.cc:824] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:36.773207 15201 log.cc:824] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:36.775032 15200 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: No bootstrap required, opened a new log
I20260620 12:27:36.775040 15201 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: No bootstrap required, opened a new log
I20260620 12:27:36.775223 15201 ts_tablet_manager.cc:1397] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:27:36.775225 15200 ts_tablet_manager.cc:1397] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 12:27:36.778352 15201 raft_consensus.cc:348] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:36.778386 15200 raft_consensus.cc:348] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:36.778699 15201 raft_consensus.cc:374] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:36.778712 15200 raft_consensus.cc:374] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:36.778777 15200 raft_consensus.cc:729] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108aadfb220241f3aa20b9c8de306c7a, State: Initialized, Role: FOLLOWER
I20260620 12:27:36.778777 15201 raft_consensus.cc:729] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73646260804422a12a4e42dcb1ee6a, State: Initialized, Role: FOLLOWER
I20260620 12:27:36.779210 15200 consensus_queue.cc:260] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:36.779210 15201 consensus_queue.cc:260] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:36.779855 15055 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260620 12:27:36.779853 15187 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260620 12:27:36.780193 15200 ts_tablet_manager.cc:1428] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 12:27:36.780194 15201 ts_tablet_manager.cc:1428] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
W20260620 12:27:36.837400 14918 tablet.cc:2369] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:27:36.963304 15188 tablet.cc:2369] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:27:37.024219 15056 tablet.cc:2369] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:27:37.094735 14923 raft_consensus.cc:482] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:37.094998 14923 raft_consensus.cc:504] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:37.096242 14923 leader_election.cc:288] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897), 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
I20260620 12:27:37.101639 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.096716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51263 (local address 127.0.98.2:32897)
0620 12:27:37.096840 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:37.096844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:37.096966 (+ 122us) server_negotiation.cc:407] Connection header received
0620 12:27:37.097151 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:37.097154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:37.097208 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:37.097297 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:37.097676 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.098453 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.100351 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.100967 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.101094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:37.101166 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:37.101431 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 12:27:37.101504 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:27:37.101637 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.096622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48413 (local address 127.0.98.3:40573)
0620 12:27:37.096758 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:37.096763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:37.096969 (+ 206us) server_negotiation.cc:407] Connection header received
0620 12:27:37.097151 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:37.097154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:37.097208 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:37.097295 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:37.097677 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.098453 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.100394 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.101051 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.101143 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:37.101200 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:37.101431 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 12:27:37.101513 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:27:37.101709 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.096487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40573 (local address 127.0.98.1:48413)
0620 12:27:37.096847 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.096862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:37.097013 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:37.097344 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:37.097346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:37.097357 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:37.097562 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:37.097567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.098576 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.098579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:37.100219 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:37.100230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.101150 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.101153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:37.101249 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:37.101340 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 12:27:37.101409 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 12:27:37.101507 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":246,"spinlock_wait_cycles":11008,"thread_start_us":94,"threads_started":1}
I20260620 12:27:37.101907 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.096512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32897 (local address 127.0.98.1:51263)
0620 12:27:37.096879 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.096893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:37.097013 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:37.097326 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:37.097330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:37.097351 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:37.097560 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:37.097567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.098576 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.098579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:37.100219 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:37.100230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.101085 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.101088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:37.101244 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:37.101302 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 12:27:37.101387 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 12:27:37.101522 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":227,"spinlock_wait_cycles":20096,"thread_start_us":76,"threads_started":1}
I20260620 12:27:37.102478 15141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7b962fb9f54c138ccb276f27af7604" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73646260804422a12a4e42dcb1ee6a" is_pre_election: true
I20260620 12:27:37.102808 15141 raft_consensus.cc:2454] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 0.
I20260620 12:27:37.103312 14803 leader_election.cc:302] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf, 8f73646260804422a12a4e42dcb1ee6a; no voters:
I20260620 12:27:37.103462 15009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7b962fb9f54c138ccb276f27af7604" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108aadfb220241f3aa20b9c8de306c7a" is_pre_election: true
I20260620 12:27:37.103590 14923 raft_consensus.cc:2781] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:27:37.103679 14923 raft_consensus.cc:482] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:27:37.103751 14923 raft_consensus.cc:3037] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:37.103767 15009 raft_consensus.cc:2454] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 0.
I20260620 12:27:37.104909 14923 raft_consensus.cc:504] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:37.105320 14923 leader_election.cc:288] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 election: Requested vote from peers 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897), 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
I20260620 12:27:37.105631 15009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7b962fb9f54c138ccb276f27af7604" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108aadfb220241f3aa20b9c8de306c7a"
I20260620 12:27:37.105723 15141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7b962fb9f54c138ccb276f27af7604" candidate_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f73646260804422a12a4e42dcb1ee6a"
I20260620 12:27:37.105760 15009 raft_consensus.cc:3037] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:37.105890 15141 raft_consensus.cc:3037] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:37.106863 15009 raft_consensus.cc:2454] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 1.
I20260620 12:27:37.106995 15141 raft_consensus.cc:2454] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d22252d3e9d4ca3a200d4872cd34ecf in term 1.
I20260620 12:27:37.107141 14806 leader_election.cc:302] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf, 108aadfb220241f3aa20b9c8de306c7a; no voters:
I20260620 12:27:37.107311 14923 raft_consensus.cc:2781] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:37.107504 14923 raft_consensus.cc:686] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 LEADER]: Becoming Leader. State: Replica: 0d22252d3e9d4ca3a200d4872cd34ecf, State: Running, Role: LEADER
I20260620 12:27:37.107753 14923 consensus_queue.cc:237] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:37.110594 14728 catalog_manager.cc:5495] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d22252d3e9d4ca3a200d4872cd34ecf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } health_report { overall_health: HEALTHY } } }
I20260620 12:27:37.122231 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.118984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.0.98.1:41681)
0620 12:27:37.119254 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:37.119261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:37.119273 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:27:37.119331 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:37.119333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:37.119378 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:37.119481 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:27:37.119687 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.120166 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.120914 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:37.121127 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:37.121203 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:37.121451 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:27:37.121843 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:27:37.121929 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:37.121986 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 12:27:37.122028 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":136,"threads_started":1}
I20260620 12:27:37.125772 15141 raft_consensus.cc:1261] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Refusing update from remote peer 0d22252d3e9d4ca3a200d4872cd34ecf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:37.125782 15009 raft_consensus.cc:1261] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Refusing update from remote peer 0d22252d3e9d4ca3a200d4872cd34ecf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:37.126449 14923 consensus_queue.cc:1035] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:37.126751 15208 consensus_queue.cc:1035] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:37.133008 15211 mvcc.cc:205] Tried to move back new op lower bound from 7298901840381677568 to 7298901840315916288. Current Snapshot: MvccSnapshot[applied={T|T < 7298901840381677568}]
I20260620 12:27:37.139241 15213 mvcc.cc:205] Tried to move back new op lower bound from 7298901840381677568 to 7298901840315916288. Current Snapshot: MvccSnapshot[applied={T|T < 7298901840381677568}]
I20260620 12:27:37.141690 14728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:37.141814 14728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:37.144055 14728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Sending DeleteTablet for 3 replicas of tablet cc7b962fb9f54c138ccb276f27af7604
I20260620 12:27:37.144616 14989 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7b962fb9f54c138ccb276f27af7604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60398
I20260620 12:27:37.144785 14851 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7b962fb9f54c138ccb276f27af7604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37274
I20260620 12:27:37.145231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14767
I20260620 12:27:37.145298 15121 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7b962fb9f54c138ccb276f27af7604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42786
I20260620 12:27:37.146914 15220 tablet_replica.cc:330] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: stopping tablet replica
I20260620 12:27:37.147208 15220 raft_consensus.cc:2229] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:37.147495 15220 raft_consensus.cc:2258] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:37.150043 15220 ts_tablet_manager.cc:1905] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:37.150696 15222 tablet_replica.cc:330] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: stopping tablet replica
I20260620 12:27:37.151007 15222 raft_consensus.cc:2229] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:37.151300 15222 raft_consensus.cc:2258] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:37.152611 15222 ts_tablet_manager.cc:1905] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:37.153184 15220 ts_tablet_manager.cc:1918] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:37.153276 15220 log.cc:1196] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/cc7b962fb9f54c138ccb276f27af7604
W20260620 12:27:37.153337 14714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41681 (error 108)
I20260620 12:27:37.153609 15220 ts_tablet_manager.cc:1939] T cc7b962fb9f54c138ccb276f27af7604 P 108aadfb220241f3aa20b9c8de306c7a: Deleting consensus metadata
I20260620 12:27:37.154270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14922
I20260620 12:27:37.154350 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.154026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41681 (local address 127.0.0.1:50682)
0620 12:27:37.154197 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.154264 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 12:27:37.154615 14714 catalog_manager.cc:4584] TS 0d22252d3e9d4ca3a200d4872cd34ecf (127.0.98.1:41681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc7b962fb9f54c138ccb276f27af7604: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41681: connect: Connection refused (error 111)
I20260620 12:27:37.155618 15222 ts_tablet_manager.cc:1918] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:37.155727 15222 log.cc:1196] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/cc7b962fb9f54c138ccb276f27af7604
I20260620 12:27:37.156045 15222 ts_tablet_manager.cc:1939] T cc7b962fb9f54c138ccb276f27af7604 P 8f73646260804422a12a4e42dcb1ee6a: Deleting consensus metadata
I20260620 12:27:37.157176 14713 catalog_manager.cc:4841] TS 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573): tablet cc7b962fb9f54c138ccb276f27af7604 (table test-table [id=dda5ebd5df4848239a896f1757e35a36]) successfully deleted
I20260620 12:27:37.161724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 15058
I20260620 12:27:37.162644 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.161834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32897 (local address 127.0.0.1:60400)
0620 12:27:37.162467 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.162539 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545}
I20260620 12:27:37.167563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 14696
I20260620 12:27:37.174021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--webserver_interface=127.0.98.62
--webserver_port=46425
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:37.281301 15224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:37.281543 15224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:37.281580 15224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:37.285660 15224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:37.285712 15224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:37.285730 15224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:37.285746 15224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:37.285761 15224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:37.291325 15224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:37.292357 15224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:37.293380 15224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:27:37.301936 15224 server_base.cc:1034] running on GCE node
W20260620 12:27:37.301891 15229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.301891 15230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.301980 15232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:37.302780 15224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:37.303902 15224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:37.305092 15224 hybrid_clock.cc:648] HybridClock initialized: now 1781958457305079 us; error 25 us; skew 500 ppm
Jun 20 12:27:37 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:37.307880 15224 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:37.308758 15224 webserver.cc:460] Webserver started at http://127.0.98.62:46425/ using document root <none> and password file <none>
I20260620 12:27:37.309319 15224 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:37.309414 15224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:37.313252 15224 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:37.315438 15239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.316640 15224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:37.316776 15224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "ad3e27a79bc54adfbf09ab08f65312f0"
format_stamp: "Formatted at 2026-06-20 12:27:34 on dist-test-slave-bwpp"
server_key: "e09ee1d88f17afea5d23625a6dd80170"
server_key_iv: "066e27ecd04756ab96750b6f7c254818"
server_key_version: "encryptionkey@0"
I20260620 12:27:37.317314 15224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:37.333730 15224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:37.336683 15224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:37.336884 15224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:37.344386 15224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40239
I20260620 12:27:37.344682 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40239 every 8 connection(s)
I20260620 12:27:37.345559 15224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:37.349726 15292 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:27:37.350351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 15224
I20260620 12:27:37.350818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41681
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40239
--webserver_port=44893
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:37.351538 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Bootstrap starting.
I20260620 12:27:37.368491 15292 log.cc:824] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:37.379442 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:27:37.379817 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: Bootstrap complete.
I20260620 12:27:37.384002 15292 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:37.384361 15292 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3e27a79bc54adfbf09ab08f65312f0, State: Initialized, Role: FOLLOWER
I20260620 12:27:37.384905 15292 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:37.385022 15292 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:27:37.385109 15292 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:27:37.385223 15292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:27:37.387084 15292 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:37.387403 15292 leader_election.cc:302] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad3e27a79bc54adfbf09ab08f65312f0; no voters:
I20260620 12:27:37.387710 15292 leader_election.cc:288] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 12:27:37.387841 15296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:27:37.388093 15296 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [term 3 LEADER]: Becoming Leader. State: Replica: ad3e27a79bc54adfbf09ab08f65312f0, State: Running, Role: LEADER
I20260620 12:27:37.388412 15296 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260620 12:27:37.388878 15292 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:37.389873 15297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260620 12:27:37.389904 15298 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad3e27a79bc54adfbf09ab08f65312f0. Latest consensus state: current_term: 3 leader_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad3e27a79bc54adfbf09ab08f65312f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260620 12:27:37.390005 15297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:37.390015 15298 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0 [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:37.394320 15311 catalog_manager.cc:1259] Loaded cluster ID: 159e4b997c794c86971b1b45c40bccde
I20260620 12:27:37.394392 15311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: loading cluster ID for follower catalog manager: success
W20260620 12:27:37.395730 15311 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ad3e27a79bc54adfbf09ab08f65312f0: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260620 12:27:37.395803 15311 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260620 12:27:37.395862 15305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:37.398572 15305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:37.399096 15305 catalog_manager.cc:1259] Loaded cluster ID: 159e4b997c794c86971b1b45c40bccde
I20260620 12:27:37.399154 15305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 12:27:37.399767 15305 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f990a37e9ad google::LogMessage::Fail() at ??:0
@ 0x7f990a38188a google::LogMessage::SendToLog() at ??:0
@ 0x7f990a37e4de google::LogMessage::Flush() at ??:0
@ 0x7f990a37fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9914cbdfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9914cbed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9914cbb579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9914cff640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f990c5d03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f990c5d0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f990c5d23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f990c5c1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f990dc056db start_thread at ??:0
@ 0x7f9908e7c61f clone at ??:0
W20260620 12:27:37.491056 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:37.491281 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:37.491334 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:37.495257 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:37.495307 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:37.495394 15294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:27:37.500516 15294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44893
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:37.501574 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:37.502579 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:37.512030 15334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.512135 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.512043 15336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:37.512341 15294 server_base.cc:1034] running on GCE node
I20260620 12:27:37.512750 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:37.513422 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:37.514612 15294 hybrid_clock.cc:648] HybridClock initialized: now 1781958457514576 us; error 47 us; skew 500 ppm
Jun 20 12:27:37 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:37.517833 15294 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:27:37.518653 15294 webserver.cc:460] Webserver started at http://127.0.98.1:44893/ using document root <none> and password file <none>
I20260620 12:27:37.519218 15294 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:37.519275 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:37.523006 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 12:27:37.525089 15343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.526358 15294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:37.526484 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "0d22252d3e9d4ca3a200d4872cd34ecf"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "7f4a6e5eb963094902276e50b7fe411e"
server_key_iv: "b1e7f76817981fc39eef642486b403fb"
server_key_version: "encryptionkey@0"
I20260620 12:27:37.527060 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:37.538249 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:37.541217 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:37.541409 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:37.541941 15294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:37.543026 15350 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:27:37.548230 15294 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:27:37.548317 15294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 12:27:37.548395 15294 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:27:37.549957 15294 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:27:37.550034 15294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 12:27:37.550335 15350 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap starting.
I20260620 12:27:37.560810 15294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41681
I20260620 12:27:37.560879 15457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41681 every 8 connection(s)
I20260620 12:27:37.561764 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:27:37.567327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 15294
I20260620 12:27:37.567761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32897
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40239
--webserver_port=40359
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:27:37.571503 15350 log.cc:824] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:37.580463 15350 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:27:37.580911 15350 tablet_bootstrap.cc:492] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Bootstrap complete.
I20260620 12:27:37.582386 15350 ts_tablet_manager.cc:1397] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20260620 12:27:37.585639 15350 raft_consensus.cc:348] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:37.586185 15350 raft_consensus.cc:729] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d22252d3e9d4ca3a200d4872cd34ecf, State: Initialized, Role: FOLLOWER
I20260620 12:27:37.586735 15350 consensus_queue.cc:260] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:37.587726 15350 ts_tablet_manager.cc:1428] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260620 12:27:37.625166 15460 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:27:37.563652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:40721)
0620 12:27:37.564118 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.564171 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:37.564947 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:37.624376 (+ 59429us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":151,"threads_started":1}
W20260620 12:27:37.625643 15458 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40239: Client connection negotiation failed: client connection to 127.0.98.62:40239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:27:37.626379 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.626141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:48457)
0620 12:27:37.626262 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.626324 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 12:27:37.627056 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.626851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:45681)
0620 12:27:37.626956 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.627010 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 12:27:37.627245 15458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:27:37.680650 15461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:37.680889 15461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:37.680931 15461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:37.685155 15461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:37.685211 15461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:37.685271 15461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:27:37.690479 15461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40359
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:37.691524 15461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:37.692507 15461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:37.701750 15469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.701792 15468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.701818 15471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:37.701978 15461 server_base.cc:1034] running on GCE node
I20260620 12:27:37.702484 15461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:37.703132 15461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:37.704309 15461 hybrid_clock.cc:648] HybridClock initialized: now 1781958457704291 us; error 30 us; skew 500 ppm
Jun 20 12:27:37 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:37.706921 15461 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:27:37.707674 15461 webserver.cc:460] Webserver started at http://127.0.98.2:40359/ using document root <none> and password file <none>
I20260620 12:27:37.708187 15461 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:37.708278 15461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:37.711908 15461 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:27:37.713904 15478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.715114 15461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:27:37.715241 15461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "108aadfb220241f3aa20b9c8de306c7a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "91a96c7fcdb6b04b68659c17e71c1c22"
server_key_iv: "ff183dcccba93262531d820ec3947087"
server_key_version: "encryptionkey@0"
I20260620 12:27:37.715761 15461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:37.729379 15461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:37.732354 15461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:37.732549 15461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:37.733124 15461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:37.733978 15461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:37.734046 15461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.734117 15461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:37.734158 15461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.743139 15461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32897
I20260620 12:27:37.743175 15591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32897 every 8 connection(s)
I20260620 12:27:37.744163 15461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:27:37.747767 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.746255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:60391)
0620 12:27:37.746968 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.747077 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":107,"threads_started":1}
W20260620 12:27:37.748252 15592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40239: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
I20260620 12:27:37.748746 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.748553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:49463)
0620 12:27:37.748654 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.748698 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 12:27:37.749418 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.749245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:60345)
0620 12:27:37.749333 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.749372 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 12:27:37.749576 15592 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:27:37.754177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 15461
I20260620 12:27:37.754520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40573
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40239
--webserver_port=46545
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:37.863070 15595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:37.863322 15595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:37.863365 15595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:37.867323 15595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:37.867378 15595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:37.867444 15595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:27:37.872551 15595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46545
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:37.873517 15595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:37.874481 15595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:37.883769 15601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.883917 15603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:37.883769 15600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:37.884073 15595 server_base.cc:1034] running on GCE node
I20260620 12:27:37.884511 15595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:37.885142 15595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:37.886324 15595 hybrid_clock.cc:648] HybridClock initialized: now 1781958457886316 us; error 28 us; skew 500 ppm
Jun 20 12:27:37 dist-test-slave-bwpp krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:37.888938 15595 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:27:37.889765 15595 webserver.cc:460] Webserver started at http://127.0.98.3:46545/ using document root <none> and password file <none>
I20260620 12:27:37.890328 15595 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:37.890420 15595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:37.894063 15595 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:37.896059 15610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.897210 15595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:27:37.897341 15595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "8f73646260804422a12a4e42dcb1ee6a"
format_stamp: "Formatted at 2026-06-20 12:27:35 on dist-test-slave-bwpp"
server_key: "03b82957c7c9e32e973fc4ad747b572d"
server_key_iv: "492449483d00e222bdc14ff142e2d77e"
server_key_version: "encryptionkey@0"
I20260620 12:27:37.897874 15595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:37.929502 15595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:37.932651 15595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:37.932798 15595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:37.933316 15595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:37.934116 15595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:37.934165 15595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.934203 15595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:37.934226 15595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:37.942798 15595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40573
I20260620 12:27:37.942879 15723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40573 every 8 connection(s)
I20260620 12:27:37.943832 15595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:27:37.947257 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.945858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:45745)
0620 12:27:37.946478 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.946578 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":116,"threads_started":1}
W20260620 12:27:37.947669 15724 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40239: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
I20260620 12:27:37.948236 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.948025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:35273)
0620 12:27:37.948136 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.948187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 12:27:37.948789 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:37.948594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:50583)
0620 12:27:37.948692 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:37.948736 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 12:27:37.948959 15724 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:27:37.951112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 15595
I20260620 12:27:38.628240 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:38.627777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:42701)
0620 12:27:38.628060 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:38.628134 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
I20260620 12:27:38.750491 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:38.750090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:34697)
0620 12:27:38.750343 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:38.750393 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260620 12:27:38.949839 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:38.949473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:48177)
0620 12:27:38.949695 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:38.949745 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260620 12:27:39.427374 15731 raft_consensus.cc:482] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:39.427608 15731 raft_consensus.cc:504] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:39.428833 15731 leader_election.cc:288] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897), 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
Jun 20 12:27:39 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 12:27:39 dist-test-slave-bwpp krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 12:27:39.442840 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:39.429115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40573 (local address 127.0.98.1:39831)
0620 12:27:39.429628 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:39.429640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:39.429719 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:39.431566 (+ 1847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:39.431571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:39.431966 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:39.432487 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:39.432502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.433810 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.433813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:39.434693 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:39.434704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.435407 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.435410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:39.435683 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:39.436431 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:39.436444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:39.438263 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:39.440730 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:39.440736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:39.440740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:39.441001 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:39.441345 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:39.441349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:39.441350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:39.441435 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:39.442065 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:39.442067 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:39.442139 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 12:27:39.442258 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 12:27:39.442580 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":311,"spinlock_wait_cycles":1792,"thread_start_us":77,"threads_started":1}
I20260620 12:27:39.442852 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:39.429136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49141 (local address 127.0.98.2:32897)
0620 12:27:39.429636 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:39.429653 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:39.429696 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:27:39.430628 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:39.430638 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:39.431139 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:39.431487 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:39.432676 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.433627 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.434825 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.435280 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.435459 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:39.438121 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:39.438127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:39.438143 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:39.438178 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:39.440066 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:39.440656 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:39.440660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:39.440664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:39.440831 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:39.441201 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:39.441204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:39.441205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:39.441703 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:39.441801 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:39.442412 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 12:27:39.442689 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":123,"threads_started":1}
I20260620 12:27:39.442871 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:39.429375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39831 (local address 127.0.98.3:40573)
0620 12:27:39.429717 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:39.429737 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:39.429767 (+ 30us) server_negotiation.cc:407] Connection header received
0620 12:27:39.430700 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:39.430709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:39.431139 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:39.431479 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:39.432672 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.433664 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.434822 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.435285 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.435459 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:39.438464 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:39.438470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:39.438484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:39.438549 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:39.440629 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:39.441101 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:39.441107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:39.441112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:39.441246 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:39.441531 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:39.441533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:39.441534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:39.441958 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:39.442086 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:39.442392 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 12:27:39.442684 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":83,"threads_started":1}
I20260620 12:27:39.442862 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:39.428992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32897 (local address 127.0.98.1:49141)
0620 12:27:39.429199 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:39.429214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:39.429388 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:39.431965 (+ 2577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:39.431968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:39.432131 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:39.432487 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:39.432503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.433769 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.433774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:39.434704 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:39.434719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:39.435405 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:39.435409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:39.435682 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:39.436369 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:39.436385 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:39.437895 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:39.440180 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:39.440185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:39.440199 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:39.440539 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:39.440932 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:39.440934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:39.440936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:39.441099 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:39.441815 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:39.441821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:39.442056 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 12:27:39.442307 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 12:27:39.442688 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":264,"spinlock_wait_cycles":2304,"thread_start_us":63,"threads_started":1}
W20260620 12:27:39.445318 15344 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
W20260620 12:27:39.445474 15347 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
I20260620 12:27:39.445609 15347 leader_election.cc:302] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf; no voters: 108aadfb220241f3aa20b9c8de306c7a, 8f73646260804422a12a4e42dcb1ee6a
I20260620 12:27:39.445850 15731 raft_consensus.cc:2726] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:27:39.629531 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:39.629217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:46009)
0620 12:27:39.629354 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:39.629432 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 12:27:39.751578 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:39.751221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:54395)
0620 12:27:39.751431 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:39.751482 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260620 12:27:39.951007 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:39.950575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:36151)
0620 12:27:39.950864 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:39.950921 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":170,"threads_started":1}
I20260620 12:27:40.630704 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:40.630264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:47813)
0620 12:27:40.630492 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:40.630546 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260620 12:27:40.752669 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:40.752252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:43999)
0620 12:27:40.752508 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:40.752561 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260620 12:27:40.952158 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:40.951716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:58557)
0620 12:27:40.951987 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:40.952047 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260620 12:27:41.508278 15741 raft_consensus.cc:482] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:41.508431 15741 raft_consensus.cc:504] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:41.509038 15741 leader_election.cc:288] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897), 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
W20260620 12:27:41.509750 15347 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
W20260620 12:27:41.509876 15344 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
I20260620 12:27:41.509969 15344 leader_election.cc:302] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf; no voters: 108aadfb220241f3aa20b9c8de306c7a, 8f73646260804422a12a4e42dcb1ee6a
I20260620 12:27:41.510136 15741 raft_consensus.cc:2726] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:27:41.631891 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:41.631440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:42353)
0620 12:27:41.631731 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:41.631792 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":79,"threads_started":1}
I20260620 12:27:41.753774 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:41.753381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:56759)
0620 12:27:41.753621 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:41.753674 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260620 12:27:41.953238 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:41.952827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:53021)
0620 12:27:41.953099 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:41.953153 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260620 12:27:42.633105 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:42.632674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:48939)
0620 12:27:42.632952 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:42.633009 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":67,"threads_started":1}
I20260620 12:27:42.754943 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:42.754491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:55119)
0620 12:27:42.754789 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:42.754851 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
I20260620 12:27:42.954530 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:42.954049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:53217)
0620 12:27:42.954357 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:42.954417 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260620 12:27:43.634301 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:43.633900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:49659)
0620 12:27:43.634146 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:43.634204 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260620 12:27:43.719249 15749 raft_consensus.cc:482] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:43.719408 15749 raft_consensus.cc:504] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:43.719839 15749 leader_election.cc:288] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897), 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
W20260620 12:27:43.720597 15347 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
W20260620 12:27:43.720706 15344 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
I20260620 12:27:43.720822 15344 leader_election.cc:302] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf; no voters: 108aadfb220241f3aa20b9c8de306c7a, 8f73646260804422a12a4e42dcb1ee6a
I20260620 12:27:43.720992 15749 raft_consensus.cc:2726] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:27:43.756059 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:43.755696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:35547)
0620 12:27:43.755918 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:43.755973 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260620 12:27:43.955619 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:43.955230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:45831)
0620 12:27:43.955481 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:43.955533 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260620 12:27:44.635517 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:44.635083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:51673)
0620 12:27:44.635378 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:44.635426 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260620 12:27:44.757292 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:44.756836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:42705)
0620 12:27:44.757149 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:44.757201 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
I20260620 12:27:44.956861 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:44.956367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:53747)
0620 12:27:44.956699 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:44.956759 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
I20260620 12:27:45.636834 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:45.636283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:41315)
0620 12:27:45.636599 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:45.636672 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":73,"threads_started":1}
I20260620 12:27:45.758337 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:45.757971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:55267)
0620 12:27:45.758199 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:45.758249 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260620 12:27:45.958079 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:45.957675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:45843)
0620 12:27:45.957932 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:45.957987 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260620 12:27:46.638062 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:46.637672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:54101)
0620 12:27:46.637920 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:46.637972 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260620 12:27:46.759553 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:46.759153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:39837)
0620 12:27:46.759419 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:46.759471 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260620 12:27:46.761961 15760 raft_consensus.cc:482] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:46.762123 15760 raft_consensus.cc:504] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:46.762581 15760 leader_election.cc:288] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897), 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
W20260620 12:27:46.763342 15344 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
W20260620 12:27:46.763442 15347 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
I20260620 12:27:46.763489 15344 leader_election.cc:302] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf; no voters: 108aadfb220241f3aa20b9c8de306c7a, 8f73646260804422a12a4e42dcb1ee6a
I20260620 12:27:46.763626 15760 raft_consensus.cc:2726] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:27:46.959359 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:46.958923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:52547)
0620 12:27:46.959193 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:46.959248 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260620 12:27:47.639400 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:47.638990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:44691)
0620 12:27:47.639252 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:47.639307 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":67,"threads_started":1}
I20260620 12:27:47.760794 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:47.760380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:43571)
0620 12:27:47.760645 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:47.760701 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
I20260620 12:27:47.960459 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:47.960079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:40581)
0620 12:27:47.960315 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:47.960369 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260620 12:27:48.640515 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:48.640137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:39891)
0620 12:27:48.640368 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:48.640418 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260620 12:27:48.761951 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:48.761516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:51867)
0620 12:27:48.761797 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:48.761856 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":73,"threads_started":1}
I20260620 12:27:48.961621 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:48.961216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:34425)
0620 12:27:48.961471 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:48.961527 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260620 12:27:49.641610 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:49.641206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:34309)
0620 12:27:49.641463 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:49.641515 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
I20260620 12:27:49.763145 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:49.762729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:54125)
0620 12:27:49.762981 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:49.763042 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260620 12:27:49.963001 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:49.962517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:54231)
0620 12:27:49.962812 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:49.962868 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":72,"threads_started":1}
I20260620 12:27:50.486769 15771 raft_consensus.cc:482] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:50.486896 15771 raft_consensus.cc:504] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "108aadfb220241f3aa20b9c8de306c7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32897 } } peers { permanent_uuid: "8f73646260804422a12a4e42dcb1ee6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40573 } } peers { permanent_uuid: "0d22252d3e9d4ca3a200d4872cd34ecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260620 12:27:50.487282 15771 leader_election.cc:288] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897), 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573)
W20260620 12:27:50.488006 15347 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 108aadfb220241f3aa20b9c8de306c7a (127.0.98.2:32897): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
W20260620 12:27:50.488150 15344 leader_election.cc:341] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f73646260804422a12a4e42dcb1ee6a (127.0.98.3:40573): Not found: Tablet not found: cc7b962fb9f54c138ccb276f27af7604
I20260620 12:27:50.488240 15344 leader_election.cc:302] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d22252d3e9d4ca3a200d4872cd34ecf; no voters: 108aadfb220241f3aa20b9c8de306c7a, 8f73646260804422a12a4e42dcb1ee6a
I20260620 12:27:50.488412 15771 raft_consensus.cc:2726] T cc7b962fb9f54c138ccb276f27af7604 P 0d22252d3e9d4ca3a200d4872cd34ecf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:27:50.642808 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:50.642339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:36677)
0620 12:27:50.642586 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:50.642639 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260620 12:27:50.764375 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:50.763960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:41977)
0620 12:27:50.764209 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:50.764265 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260620 12:27:50.964180 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:50.963759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:58733)
0620 12:27:50.964027 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:50.964085 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260620 12:27:51.644052 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:51.643623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:54679)
0620 12:27:51.643886 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:51.643955 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260620 12:27:51.765727 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:51.765222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:32865)
0620 12:27:51.765581 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:51.765631 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":151,"threads_started":1}
I20260620 12:27:51.965572 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:51.965141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:36763)
0620 12:27:51.965414 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:51.965472 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20260620 12:27:52.645396 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:52.644990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:58975)
0620 12:27:52.645240 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:52.645297 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260620 12:27:52.767145 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:52.766744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:54507)
0620 12:27:52.766998 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:52.767052 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260620 12:27:52.954835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 15294
I20260620 12:27:52.960580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 15461
I20260620 12:27:52.965850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 15595
2026-06-20T12:27:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19332 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:27:53 dist-test-slave-bwpp krb5kdc[15782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:27:53 dist-test-slave-bwpp krb5kdc[15782](info): set up 2 sockets
Jun 20 12:27:53 dist-test-slave-bwpp krb5kdc[15782](info): commencing operation
krb5kdc: starting...
W20260620 12:27:54.010924 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:27:54.128901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:54.236876 15795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:54.237118 15795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:54.237155 15795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:54.241137 15795 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:27:54.241183 15795 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:27:54.241199 15795 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:27:54.241212 15795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:27:54.241230 15795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:54.241276 15795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:54.241309 15795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:27:54.241329 15795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:27:54.246394 15795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:54.247397 15795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:54.248382 15795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:54.256717 15800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:54.256989 15795 server_base.cc:1034] running on GCE node
W20260620 12:27:54.256853 15801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:54.256721 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:54.257714 15795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:54.258879 15795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:54.260068 15795 hybrid_clock.cc:648] HybridClock initialized: now 1781958474260058 us; error 26 us; skew 500 ppm
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:54.262799 15795 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:27:54.263701 15795 webserver.cc:460] Webserver started at http://127.0.98.62:39893/ using document root <none> and password file <none>
I20260620 12:27:54.264259 15795 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:54.264348 15795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:54.264559 15795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:54.266541 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "a9e72ea8ce60416db6469e299e29bbdd"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "07c999ac196bff5f713ca287c32d1f4b"
server_key_iv: "ba5176c62ac444ab0bfc4758a643af6b"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.267154 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9e72ea8ce60416db6469e299e29bbdd"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "07c999ac196bff5f713ca287c32d1f4b"
server_key_iv: "ba5176c62ac444ab0bfc4758a643af6b"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.270854 15795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:27:54.273257 15810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.274446 15795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:27:54.274585 15795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "a9e72ea8ce60416db6469e299e29bbdd"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "07c999ac196bff5f713ca287c32d1f4b"
server_key_iv: "ba5176c62ac444ab0bfc4758a643af6b"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.274739 15795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:54.291013 15795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:54.293933 15795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:54.294155 15795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:54.301489 15795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46423
I20260620 12:27:54.301776 15862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46423 every 8 connection(s)
I20260620 12:27:54.302449 15795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:27:54.304507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 15795
I20260620 12:27:54.304648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:27:54.304935 392 external_mini_cluster.cc:1428] Setting key 2de3b3863341d5755b1688ade9073561
I20260620 12:27:54.305388 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:54.312520 15863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Bootstrap starting.
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:54.315083 15863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:54.315969 15863 log.cc:824] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:54.318048 15863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: No bootstrap required, opened a new log
I20260620 12:27:54.319517 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:54.306475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.0.98.62:46423)
0620 12:27:54.306929 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:54.306949 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:54.306989 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:27:54.307767 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:54.307790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:54.308214 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:54.308586 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:54.309212 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.310732 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.311551 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.312202 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.312365 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:54.314959 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:54.314965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:54.314979 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:54.315005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:54.316686 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:54.317134 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:54.317138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:54.317143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:54.317247 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:54.317530 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:54.317532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:54.317533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:54.317885 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:54.318026 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:54.318344 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 12:27:54.318644 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":164,"threads_started":1}
I20260620 12:27:54.322118 15863 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:27:54.322357 15863 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:54.322417 15863 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e72ea8ce60416db6469e299e29bbdd, State: Initialized, Role: FOLLOWER
I20260620 12:27:54.322862 15863 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:27:54.323012 15863 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:27:54.323091 15863 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:27:54.323185 15863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:54.323997 15863 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:27:54.324322 15863 leader_election.cc:302] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e72ea8ce60416db6469e299e29bbdd; no voters:
I20260620 12:27:54.324637 15863 leader_election.cc:288] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:27:54.324703 15868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:54.324992 15868 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 LEADER]: Becoming Leader. State: Replica: a9e72ea8ce60416db6469e299e29bbdd, State: Running, Role: LEADER
I20260620 12:27:54.325327 15868 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:27:54.325786 15863 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:27:54.326875 15870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e72ea8ce60416db6469e299e29bbdd. Latest consensus state: current_term: 1 leader_uuid: "a9e72ea8ce60416db6469e299e29bbdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260620 12:27:54.327009 15870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:54.327450 15877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:27:54.328943 15869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e72ea8ce60416db6469e299e29bbdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260620 12:27:54.329044 15869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: This master's current role is: LEADER
I20260620 12:27:54.330523 15877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:27:54.335570 15877 catalog_manager.cc:1347] Generated new cluster ID: 8d9747ac42a94b2b804fd3e6289d4c0c
I20260620 12:27:54.335630 15877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:27:54.347919 15877 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:27:54.348846 15877 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:27:54.356254 15877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Generated new TSK 0
I20260620 12:27:54.357085 15877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:27:54.395267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46423
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:54.503618 15889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:54.503854 15889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:54.503894 15889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:54.507833 15889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:54.507879 15889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:54.507993 15889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:27:54.513072 15889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:54.514045 15889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:54.515043 15889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:54.524247 15894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:54.524292 15897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:54.524427 15889 server_base.cc:1034] running on GCE node
W20260620 12:27:54.524556 15895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:54.524996 15889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:54.525636 15889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:54.526850 15889 hybrid_clock.cc:648] HybridClock initialized: now 1781958474526835 us; error 32 us; skew 500 ppm
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:54.529677 15889 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:27:54.530442 15889 webserver.cc:460] Webserver started at http://127.0.98.1:33719/ using document root <none> and password file <none>
I20260620 12:27:54.530997 15889 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:54.531056 15889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:54.531205 15889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:54.533035 15889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a7a764ab60844f9b4b011634b529269"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "daf4eabb031a6a6e70987c5f8973766d"
server_key_iv: "d332fb7d50269608d7caf038a0bf89db"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.533514 15889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a7a764ab60844f9b4b011634b529269"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "daf4eabb031a6a6e70987c5f8973766d"
server_key_iv: "d332fb7d50269608d7caf038a0bf89db"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.536916 15889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:27:54.539130 15904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.540243 15889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:27:54.540333 15889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "6a7a764ab60844f9b4b011634b529269"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "daf4eabb031a6a6e70987c5f8973766d"
server_key_iv: "d332fb7d50269608d7caf038a0bf89db"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.540454 15889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:54.552584 15889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:54.555828 15889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:54.555979 15889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:54.556514 15889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:54.557302 15889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:54.557349 15889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.557417 15889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:54.557448 15889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.566028 15889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46325
I20260620 12:27:54.566058 16017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46325 every 8 connection(s)
I20260620 12:27:54.566905 15889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:27:54.570941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 15889
I20260620 12:27:54.571081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:27:54.571388 392 external_mini_cluster.cc:1428] Setting key f0dec091293040445ab25675a3595c47
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:27:54.581354 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:54.568655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53119 (local address 127.0.98.62:46423)
0620 12:27:54.568916 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:54.568921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:54.569610 (+ 689us) server_negotiation.cc:407] Connection header received
0620 12:27:54.570934 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:54.570938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:54.570998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:54.571109 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:54.572429 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.573104 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.574873 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.575109 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.575225 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:54.577760 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:54.577763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:54.577765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:54.577788 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:54.579346 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:54.579862 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:54.579866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:54.579868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:54.579947 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:54.580284 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:54.580289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:54.580291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:54.580482 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:54.580585 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:54.581067 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 12:27:54.581191 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260620 12:27:54.582217 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:54.569005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:53119)
0620 12:27:54.569450 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:54.569501 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:54.570723 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:54.571312 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:54.571319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:54.571721 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:54.572285 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:54.572299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.573241 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.573248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:54.574735 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:54.574746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.575261 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.575265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:54.575455 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:54.576053 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:54.576069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:54.577582 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:54.579479 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:54.579485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:54.579500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:54.579752 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:54.580040 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:54.580044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:54.580046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:54.580188 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:54.580598 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:54.580606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:54.580842 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 12:27:54.581028 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 12:27:54.581394 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:27:54.583479 16018 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46423
I20260620 12:27:54.583717 16018 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:54.584270 16018 heartbeater.cc:511] Master 127.0.98.62:46423 requested a full tablet report, sending...
I20260620 12:27:54.585824 15827 ts_manager.cc:194] Registered new tserver with Master: 6a7a764ab60844f9b4b011634b529269 (127.0.98.1:46325)
I20260620 12:27:54.587069 15827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:27:54.597998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46423
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:54.708067 16023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:54.708367 16023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:54.708451 16023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:54.712886 16023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:54.712982 16023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:54.713079 16023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:27:54.718242 16023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:54.719298 16023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:54.720407 16023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:54.729956 16031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:54.729946 16029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:54.729950 16028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:54.730290 16023 server_base.cc:1034] running on GCE node
I20260620 12:27:54.730791 16023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:54.731408 16023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:54.732607 16023 hybrid_clock.cc:648] HybridClock initialized: now 1781958474732597 us; error 32 us; skew 500 ppm
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:54.735193 16023 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:27:54.736065 16023 webserver.cc:460] Webserver started at http://127.0.98.2:34773/ using document root <none> and password file <none>
I20260620 12:27:54.736621 16023 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:54.736709 16023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:54.736935 16023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:54.739014 16023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d156f39eefa46dbac4936e1fbf64662"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "ece1a4a09166c5de8a151f69a73ef867"
server_key_iv: "0fcbf943f4fd2b0484f3ee2ba9700d4c"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.739593 16023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d156f39eefa46dbac4936e1fbf64662"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "ece1a4a09166c5de8a151f69a73ef867"
server_key_iv: "0fcbf943f4fd2b0484f3ee2ba9700d4c"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.743491 16023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:27:54.746037 16038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.747316 16023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 12:27:54.747433 16023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "3d156f39eefa46dbac4936e1fbf64662"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "ece1a4a09166c5de8a151f69a73ef867"
server_key_iv: "0fcbf943f4fd2b0484f3ee2ba9700d4c"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.747535 16023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:54.777066 16023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:54.780638 16023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:54.780786 16023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:54.781325 16023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:54.782087 16023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:54.782133 16023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.782171 16023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:54.782195 16023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.790681 16023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42733
I20260620 12:27:54.790752 16151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42733 every 8 connection(s)
I20260620 12:27:54.791575 16023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:27:54.794169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16023
I20260620 12:27:54.794304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:27:54.794617 392 external_mini_cluster.cc:1428] Setting key c6cb8e8abb4ceff4a03f35438d14d24d
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:27:54.806382 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:54.793280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47285 (local address 127.0.98.62:46423)
0620 12:27:54.793411 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:54.793415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:54.794251 (+ 836us) server_negotiation.cc:407] Connection header received
0620 12:27:54.795840 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:54.795843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:54.795915 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:54.796007 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:54.797174 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.797987 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.799813 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.800028 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.800107 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:54.802887 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:54.802890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:54.802892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:54.802914 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:54.804287 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:54.804794 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:54.804798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:54.804799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:54.804878 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:54.805248 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:54.805251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:54.805253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:54.805430 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:54.805537 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:54.806086 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 12:27:54.806203 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:27:54.807648 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:54.793682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:47285)
0620 12:27:54.794108 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:54.794148 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:54.795626 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:54.796227 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:54.796234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:54.796567 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:54.797030 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:54.797043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.798112 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.798119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:54.799684 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:54.799699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:54.800183 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:54.800190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:54.800419 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:54.801028 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:54.801054 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:54.802726 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:54.804432 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:54.804437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:54.804452 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:54.804690 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:54.804976 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:54.804979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:54.804981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:54.805149 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:54.805534 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:54.805540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:54.805766 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 12:27:54.806029 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 12:27:54.806371 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260620 12:27:54.808954 16152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46423
I20260620 12:27:54.809185 16152 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:54.809649 16152 heartbeater.cc:511] Master 127.0.98.62:46423 requested a full tablet report, sending...
I20260620 12:27:54.810751 15827 ts_manager.cc:194] Registered new tserver with Master: 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733)
I20260620 12:27:54.811359 15827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:27:54.821954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46423
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:27:54.931427 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:27:54.931708 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:27:54.931780 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:27:54.935750 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:27:54.935827 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:27:54.935915 16157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:27:54.940999 16157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:27:54.941946 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:27:54.942951 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:27:54.952642 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:54.952651 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:27:54.952659 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:27:54.952831 16157 server_base.cc:1034] running on GCE node
I20260620 12:27:54.953435 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:27:54.954088 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:27:54.955272 16157 hybrid_clock.cc:648] HybridClock initialized: now 1781958474955257 us; error 32 us; skew 500 ppm
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:27:54.957923 16157 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:27:54.958870 16157 webserver.cc:460] Webserver started at http://127.0.98.3:42677/ using document root <none> and password file <none>
I20260620 12:27:54.959386 16157 fs_manager.cc:359] Metadata directory not provided
I20260620 12:27:54.959442 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:27:54.959610 16157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:27:54.961555 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "efc0c5775e8e41f2995f16ed8c6e83a5"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "195356b887faee835a61aa249c39acf5"
server_key_iv: "5ed3034f196388ac4b3e88863379869e"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.962047 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "efc0c5775e8e41f2995f16ed8c6e83a5"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "195356b887faee835a61aa249c39acf5"
server_key_iv: "5ed3034f196388ac4b3e88863379869e"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.965633 16157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:27:54.968041 16172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.969233 16157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:27:54.969329 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "efc0c5775e8e41f2995f16ed8c6e83a5"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "195356b887faee835a61aa249c39acf5"
server_key_iv: "5ed3034f196388ac4b3e88863379869e"
server_key_version: "encryptionkey@0"
I20260620 12:27:54.969475 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:27:54.980595 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:27:54.983515 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:27:54.983717 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:27:54.984283 16157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:27:54.985086 16157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:27:54.985131 16157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.985198 16157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:27:54.985230 16157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:27:54.994477 16157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40829
I20260620 12:27:54.994544 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40829 every 8 connection(s)
I20260620 12:27:54.995364 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:27:54.998098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16157
I20260620 12:27:54.998221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:27:54.998550 392 external_mini_cluster.cc:1428] Setting key 33797c92add0c4a9704b800eb61386df
Jun 20 12:27:54 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:27:55.008781 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:54.997119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59267 (local address 127.0.98.62:46423)
0620 12:27:54.997256 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:54.997260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:54.998036 (+ 776us) server_negotiation.cc:407] Connection header received
0620 12:27:54.999628 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:54.999631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:54.999687 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:54.999792 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:55.000931 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.001464 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.002643 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.002907 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.003036 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.005412 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:55.005415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:55.005417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:55.005441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:55.006728 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:55.007214 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:55.007219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:55.007221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:55.007309 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:55.007662 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:55.007665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:55.007666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:55.007892 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:55.008013 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.008477 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.008612 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:27:55.009603 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:54.997410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:59267)
0620 12:27:54.997875 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:54.997915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:54.999352 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:54.999965 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:54.999971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:55.000317 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:27:55.000784 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.000797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.001588 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.001592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.002515 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.002525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.003069 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.003072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.003248 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.003849 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:27:55.003864 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:27:55.005224 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:27:55.006847 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:55.006853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:55.006867 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:55.007113 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:55.007401 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:27:55.007406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:27:55.007408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:27:55.007549 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:27:55.007999 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:27:55.008006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:27:55.008248 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 12:27:55.008426 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 12:27:55.008747 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
I20260620 12:27:55.010771 16286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46423
I20260620 12:27:55.011015 16286 heartbeater.cc:464] Registering TS with master...
I20260620 12:27:55.011497 16286 heartbeater.cc:511] Master 127.0.98.62:46423 requested a full tablet report, sending...
I20260620 12:27:55.012502 15827 ts_manager.cc:194] Registered new tserver with Master: efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
I20260620 12:27:55.013070 15827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59267
I20260620 12:27:55.023872 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:27:55.033589 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.025862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.0.98.62:46423)
0620 12:27:55.025991 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:55.025994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:55.026235 (+ 241us) server_negotiation.cc:407] Connection header received
0620 12:27:55.026410 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:55.026412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:55.026450 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:55.026577 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:27:55.027363 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.028253 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.029041 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.029255 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.029333 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.030738 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:27:55.030741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:27:55.030743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:27:55.030765 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:27:55.032073 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:55.032508 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:55.032510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:55.032511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:55.032583 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:27:55.032817 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:27:55.032821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:27:55.032822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:27:55.033050 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:27:55.033154 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.033377 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.033476 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:27:55.036211 15827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:27:55.039274 15827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:27:55.053905 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.048885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.0.98.2:42733)
0620 12:27:55.049238 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:55.049251 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:55.049569 (+ 318us) server_negotiation.cc:407] Connection header received
0620 12:27:55.049638 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:55.049642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:55.049797 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:55.049936 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:55.050187 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.051347 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.052572 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.053232 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.053339 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.053486 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.053648 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.053744 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":126,"threads_started":1}
I20260620 12:27:55.054422 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.048442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42733 (local address 127.0.0.1:33728)
0620 12:27:55.049381 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:55.049393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:55.049482 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:55.049936 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:55.049939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:55.049966 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:55.050075 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.050081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.051492 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.051495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.052445 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.052455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.053349 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.053352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.053456 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.053494 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 12:27:55.053602 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:27:55.053722 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":23,"thread_start_us":94,"threads_started":1}
I20260620 12:27:55.054606 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.047683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46325 (local address 127.0.0.1:48112)
0620 12:27:55.048917 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:55.048954 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:55.049105 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:55.049672 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:55.049677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:55.049699 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:55.049852 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.049859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.051322 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.051325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.052836 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.052846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.054140 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.054144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.054239 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.054324 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 12:27:55.054403 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:27:55.054473 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":114,"thread_start_us":96,"threads_started":1}
I20260620 12:27:55.055025 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.047821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.0.98.1:46325)
0620 12:27:55.048661 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:55.048679 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:55.049057 (+ 378us) server_negotiation.cc:407] Connection header received
0620 12:27:55.049291 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:55.049299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:55.049518 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:55.049675 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:55.050022 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.051164 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.052974 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.053804 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.053907 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.054078 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.054775 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.054871 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":91,"threads_started":1}
I20260620 12:27:55.055542 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.048283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40829 (local address 127.0.0.1:54140)
0620 12:27:55.049279 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:55.049292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:55.049386 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:55.050027 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:55.050030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:55.050047 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:55.050174 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.050181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.051861 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.051864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.053032 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.053039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.054235 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.054242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.054351 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.054381 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:27:55.054449 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:27:55.054534 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":8,"thread_start_us":111,"threads_started":1}
I20260620 12:27:55.055918 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.048487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.0.98.3:40829)
0620 12:27:55.048866 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:55.048878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:55.049396 (+ 518us) server_negotiation.cc:407] Connection header received
0620 12:27:55.049493 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:55.049499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:55.049692 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:55.049833 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:55.050307 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.051658 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.053144 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.053935 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.054035 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.054169 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.055716 (+ 1547us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.055816 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":169,"threads_started":1}
I20260620 12:27:55.057713 16086 tablet_service.cc:1467] Processing CreateTablet for tablet 6e39be7df31842d2bb0f65b0f9b7d55b (DEFAULT_TABLE table=test-table [id=3a47237531f54a109ff2e6423df9885f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:55.057713 15952 tablet_service.cc:1467] Processing CreateTablet for tablet 6e39be7df31842d2bb0f65b0f9b7d55b (DEFAULT_TABLE table=test-table [id=3a47237531f54a109ff2e6423df9885f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:55.058869 15952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e39be7df31842d2bb0f65b0f9b7d55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:55.058869 16086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e39be7df31842d2bb0f65b0f9b7d55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:55.065055 16220 tablet_service.cc:1467] Processing CreateTablet for tablet 6e39be7df31842d2bb0f65b0f9b7d55b (DEFAULT_TABLE table=test-table [id=3a47237531f54a109ff2e6423df9885f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:27:55.065975 16220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e39be7df31842d2bb0f65b0f9b7d55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:27:55.066210 16304 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Bootstrap starting.
I20260620 12:27:55.068523 16304 tablet_bootstrap.cc:654] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:55.068817 16303 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Bootstrap starting.
I20260620 12:27:55.069882 16304 log.cc:824] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:55.071090 16303 tablet_bootstrap.cc:654] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:55.072022 16304 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: No bootstrap required, opened a new log
I20260620 12:27:55.072022 16303 log.cc:824] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:55.072216 16304 ts_tablet_manager.cc:1397] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:27:55.073661 16307 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Bootstrap starting.
I20260620 12:27:55.074180 16303 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: No bootstrap required, opened a new log
I20260620 12:27:55.074378 16303 ts_tablet_manager.cc:1397] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 12:27:55.076156 16307 tablet_bootstrap.cc:654] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Neither blocks nor log segments found. Creating new log.
I20260620 12:27:55.076195 16304 raft_consensus.cc:348] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.076470 16304 raft_consensus.cc:374] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:55.076550 16304 raft_consensus.cc:729] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7a764ab60844f9b4b011634b529269, State: Initialized, Role: FOLLOWER
I20260620 12:27:55.076967 16307 log.cc:824] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:27:55.077005 16304 consensus_queue.cc:260] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.077664 16018 heartbeater.cc:503] Master 127.0.98.62:46423 was elected leader, sending a full tablet report...
I20260620 12:27:55.077993 16304 ts_tablet_manager.cc:1428] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:27:55.078054 16303 raft_consensus.cc:348] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.078388 16303 raft_consensus.cc:374] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:55.078476 16303 raft_consensus.cc:729] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d156f39eefa46dbac4936e1fbf64662, State: Initialized, Role: FOLLOWER
I20260620 12:27:55.078955 16303 consensus_queue.cc:260] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.079350 16307 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: No bootstrap required, opened a new log
I20260620 12:27:55.079567 16307 ts_tablet_manager.cc:1397] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 12:27:55.079648 16152 heartbeater.cc:503] Master 127.0.98.62:46423 was elected leader, sending a full tablet report...
I20260620 12:27:55.079959 16303 ts_tablet_manager.cc:1428] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:27:55.083242 16307 raft_consensus.cc:348] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.083461 16307 raft_consensus.cc:374] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:27:55.083519 16307 raft_consensus.cc:729] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc0c5775e8e41f2995f16ed8c6e83a5, State: Initialized, Role: FOLLOWER
I20260620 12:27:55.083963 16307 consensus_queue.cc:260] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.084578 16286 heartbeater.cc:503] Master 127.0.98.62:46423 was elected leader, sending a full tablet report...
I20260620 12:27:55.084748 16307 ts_tablet_manager.cc:1428] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260620 12:27:55.246320 16287 tablet.cc:2369] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:27:55.264670 16308 raft_consensus.cc:482] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:27:55.264947 16308 raft_consensus.cc:504] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.266165 16308 leader_election.cc:288] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
I20260620 12:27:55.270129 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.266406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42733 (local address 127.0.98.1:57825)
0620 12:27:55.266731 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:55.266746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:55.266879 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:55.267183 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:55.267189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:55.267209 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:55.267347 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.267354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.268083 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.268087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.269006 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.269015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.269676 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.269683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.269788 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.269816 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:27:55.269887 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 12:27:55.269945 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":56,"thread_start_us":93,"threads_started":1}
I20260620 12:27:55.270175 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.266507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57825 (local address 127.0.98.2:42733)
0620 12:27:55.266721 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:55.266725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:55.266848 (+ 123us) server_negotiation.cc:407] Connection header received
0620 12:27:55.267017 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:55.267022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:55.267083 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:55.267143 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:55.267453 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.267964 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.269126 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.269550 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.269632 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.269801 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.269907 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.270010 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 12:27:55.270373 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.266542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45989 (local address 127.0.98.3:40829)
0620 12:27:55.266786 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:55.266792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:55.266917 (+ 125us) server_negotiation.cc:407] Connection header received
0620 12:27:55.267082 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:55.267085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:55.267150 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:55.267227 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:27:55.267491 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.267967 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.269168 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.269627 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.269730 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.269798 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.270141 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.270234 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260620 12:27:55.270414 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.266414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40829 (local address 127.0.98.1:45989)
0620 12:27:55.266796 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:55.266829 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 12:27:55.266959 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:27:55.267254 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:27:55.267257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:27:55.267273 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:27:55.267370 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.267374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.268080 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.268085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.269014 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:27:55.269023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.269833 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.269838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:27:55.269972 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.270017 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 12:27:55.270110 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 12:27:55.270181 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":62,"thread_start_us":113,"threads_started":1}
I20260620 12:27:55.271159 16240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e39be7df31842d2bb0f65b0f9b7d55b" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" is_pre_election: true
I20260620 12:27:55.271159 16106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e39be7df31842d2bb0f65b0f9b7d55b" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d156f39eefa46dbac4936e1fbf64662" is_pre_election: true
I20260620 12:27:55.271493 16106 raft_consensus.cc:2454] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 0.
I20260620 12:27:55.271493 16240 raft_consensus.cc:2454] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 0.
I20260620 12:27:55.271981 15907 leader_election.cc:302] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d156f39eefa46dbac4936e1fbf64662, 6a7a764ab60844f9b4b011634b529269; no voters:
I20260620 12:27:55.272210 16308 raft_consensus.cc:2781] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:27:55.272291 16308 raft_consensus.cc:482] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:27:55.272322 16308 raft_consensus.cc:3037] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:55.273499 16308 raft_consensus.cc:504] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.273841 16308 leader_election.cc:288] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 election: Requested vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
I20260620 12:27:55.274333 16106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e39be7df31842d2bb0f65b0f9b7d55b" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d156f39eefa46dbac4936e1fbf64662"
I20260620 12:27:55.274333 16240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e39be7df31842d2bb0f65b0f9b7d55b" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5"
I20260620 12:27:55.274458 16240 raft_consensus.cc:3037] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:55.274458 16106 raft_consensus.cc:3037] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:27:55.275765 16240 raft_consensus.cc:2454] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 1.
I20260620 12:27:55.275776 16106 raft_consensus.cc:2454] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 1.
I20260620 12:27:55.276087 15905 leader_election.cc:302] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d156f39eefa46dbac4936e1fbf64662, 6a7a764ab60844f9b4b011634b529269, efc0c5775e8e41f2995f16ed8c6e83a5; no voters:
I20260620 12:27:55.276310 16308 raft_consensus.cc:2781] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:27:55.276546 16308 raft_consensus.cc:686] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7a764ab60844f9b4b011634b529269, State: Running, Role: LEADER
I20260620 12:27:55.276911 16308 consensus_queue.cc:237] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:27:55.280215 15827 catalog_manager.cc:5495] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7a764ab60844f9b4b011634b529269 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6a7a764ab60844f9b4b011634b529269" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } health_report { overall_health: UNKNOWN } } }
W20260620 12:27:55.292646 16153 tablet.cc:2369] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:27:55.318308 16019 tablet.cc:2369] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:27:55.342126 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.338568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.0.98.1:46325)
0620 12:27:55.338737 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:27:55.338742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:27:55.338758 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:27:55.338929 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:27:55.338932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:27:55.338988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:27:55.339101 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:27:55.339360 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.339862 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.340721 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:27:55.340944 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:27:55.341053 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:27:55.341354 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:27:55.341756 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:27:55.341889 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 12:27:55.341931 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 12:27:55.341982 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:27:55.347836 16240 raft_consensus.cc:1261] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Refusing update from remote peer 6a7a764ab60844f9b4b011634b529269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:55.347848 16106 raft_consensus.cc:1261] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Refusing update from remote peer 6a7a764ab60844f9b4b011634b529269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:27:55.348516 16314 consensus_queue.cc:1035] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:55.348719 16308 consensus_queue.cc:1035] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:27:55.358223 16318 mvcc.cc:205] Tried to move back new op lower bound from 7298901915017949184 to 7298901914737319936. Current Snapshot: MvccSnapshot[applied={T|T < 7298901915017949184}]
I20260620 12:27:55.358538 16319 mvcc.cc:205] Tried to move back new op lower bound from 7298901915017949184 to 7298901914737319936. Current Snapshot: MvccSnapshot[applied={T|T < 7298901915017949184}]
I20260620 12:27:55.363962 15826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:55.364123 15826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:27:55.366365 15826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Sending DeleteTablet for 3 replicas of tablet 6e39be7df31842d2bb0f65b0f9b7d55b
I20260620 12:27:55.367159 16220 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e39be7df31842d2bb0f65b0f9b7d55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54140
I20260620 12:27:55.367266 16086 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e39be7df31842d2bb0f65b0f9b7d55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33728
I20260620 12:27:55.367254 15952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e39be7df31842d2bb0f65b0f9b7d55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48112
I20260620 12:27:55.367661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 15889
I20260620 12:27:55.367702 16328 tablet_replica.cc:330] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: stopping tablet replica
I20260620 12:27:55.367729 16326 tablet_replica.cc:330] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: stopping tablet replica
I20260620 12:27:55.367982 16326 raft_consensus.cc:2229] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:55.368002 16328 raft_consensus.cc:2229] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:27:55.368237 16326 raft_consensus.cc:2258] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:55.368602 16327 tablet_replica.cc:330] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: stopping tablet replica
I20260620 12:27:55.368865 16327 raft_consensus.cc:2229] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:27:55.369143 16327 raft_consensus.cc:2258] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:27:55.369617 16326 ts_tablet_manager.cc:1905] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:55.370363 16327 ts_tablet_manager.cc:1905] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:27:55.372356 16326 ts_tablet_manager.cc:1918] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:55.372516 16326 log.cc:1196] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/6e39be7df31842d2bb0f65b0f9b7d55b
I20260620 12:27:55.372918 16326 ts_tablet_manager.cc:1939] T 6e39be7df31842d2bb0f65b0f9b7d55b P efc0c5775e8e41f2995f16ed8c6e83a5: Deleting consensus metadata
I20260620 12:27:55.373064 16327 ts_tablet_manager.cc:1918] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:27:55.373185 16327 log.cc:1196] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/6e39be7df31842d2bb0f65b0f9b7d55b
I20260620 12:27:55.373483 16327 ts_tablet_manager.cc:1939] T 6e39be7df31842d2bb0f65b0f9b7d55b P 3d156f39eefa46dbac4936e1fbf64662: Deleting consensus metadata
I20260620 12:27:55.374189 15811 catalog_manager.cc:4841] TS efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829): tablet 6e39be7df31842d2bb0f65b0f9b7d55b (table test-table [id=3a47237531f54a109ff2e6423df9885f]) successfully deleted
I20260620 12:27:55.374451 15813 catalog_manager.cc:4841] TS 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733): tablet 6e39be7df31842d2bb0f65b0f9b7d55b (table test-table [id=3a47237531f54a109ff2e6423df9885f]) successfully deleted
W20260620 12:27:55.375820 15813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46325 (error 108)
I20260620 12:27:55.375859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 16023
I20260620 12:27:55.377560 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:27:55.377197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46325 (local address 127.0.0.1:48144)
0620 12:27:55.377362 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 12:27:55.377443 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 12:27:55.377713 15813 catalog_manager.cc:4584] TS 6a7a764ab60844f9b4b011634b529269 (127.0.98.1:46325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e39be7df31842d2bb0f65b0f9b7d55b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46325: connect: Connection refused (error 111)
I20260620 12:27:55.381531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 16157
I20260620 12:27:55.386876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 15795
I20260620 12:28:15.393311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--webserver_interface=127.0.98.62
--webserver_port=39893
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:15.501365 16330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:15.501598 16330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:15.501636 16330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:15.505641 16330 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:28:15.505689 16330 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:28:15.505705 16330 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:28:15.505719 16330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:28:15.505738 16330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:15.505801 16330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:15.505831 16330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:28:15.505863 16330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:28:15.510958 16330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:15.511932 16330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:15.512915 16330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:15.526005 16337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:15.526026 16336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:15.526037 16339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:15.526103 16330 server_base.cc:1034] running on GCE node
I20260620 12:28:15.526935 16330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:15.528033 16330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:15.529215 16330 hybrid_clock.cc:648] HybridClock initialized: now 1781958495529196 us; error 33 us; skew 500 ppm
Jun 20 12:28:15 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:15.535476 16330 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:28:15.536382 16330 webserver.cc:460] Webserver started at http://127.0.98.62:39893/ using document root <none> and password file <none>
I20260620 12:28:15.536952 16330 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:15.537026 16330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:15.540804 16330 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 12:28:15.542929 16346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:15.544112 16330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:15.544252 16330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "a9e72ea8ce60416db6469e299e29bbdd"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "07c999ac196bff5f713ca287c32d1f4b"
server_key_iv: "ba5176c62ac444ab0bfc4758a643af6b"
server_key_version: "encryptionkey@0"
I20260620 12:28:15.544838 16330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:15.564190 16330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:15.567073 16330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:15.567288 16330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:15.574761 16330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46423
I20260620 12:28:15.575058 16398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46423 every 8 connection(s)
I20260620 12:28:15.575856 16330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:28:15.579420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16330
I20260620 12:28:15.579917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46325
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46423
--webserver_port=33719
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:28:15.580538 16399 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:28:15.582527 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Bootstrap starting.
I20260620 12:28:15.599371 16399 log.cc:824] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:15.610301 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:28:15.610767 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Bootstrap complete.
I20260620 12:28:15.616477 16399 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:15.617009 16399 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e72ea8ce60416db6469e299e29bbdd, State: Initialized, Role: FOLLOWER
I20260620 12:28:15.617558 16399 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:15.617704 16399 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:28:15.617779 16399 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:28:15.617872 16399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:28:15.620386 16399 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:15.620779 16399 leader_election.cc:302] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e72ea8ce60416db6469e299e29bbdd; no voters:
I20260620 12:28:15.621147 16399 leader_election.cc:288] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:28:15.621284 16403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:28:15.621567 16403 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 LEADER]: Becoming Leader. State: Replica: a9e72ea8ce60416db6469e299e29bbdd, State: Running, Role: LEADER
I20260620 12:28:15.621906 16403 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:15.622368 16399 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:28:15.623219 16404 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9e72ea8ce60416db6469e299e29bbdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260620 12:28:15.623345 16404 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:15.624122 16405 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e72ea8ce60416db6469e299e29bbdd. Latest consensus state: current_term: 2 leader_uuid: "a9e72ea8ce60416db6469e299e29bbdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260620 12:28:15.624183 16412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:28:15.624207 16405 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:15.628839 16412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:28:15.629457 16412 catalog_manager.cc:1259] Loaded cluster ID: 8d9747ac42a94b2b804fd3e6289d4c0c
I20260620 12:28:15.629516 16412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:28:15.631999 16412 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:28:15.632680 16412 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Loaded TSK: 0
I20260620 12:28:15.655946 16412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Generated new TSK 1
I20260620 12:28:15.656203 16412 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:28:15.700089 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:15.700369 16401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:15.700455 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:15.704694 16401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:15.704787 16401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:15.704873 16401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:28:15.710325 16401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33719
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:15.711395 16401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:15.712407 16401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:15.722106 16430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:15.722110 16427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:15.722312 16428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:15.723117 16401 server_base.cc:1034] running on GCE node
I20260620 12:28:15.723630 16401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:15.724434 16401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:15.725638 16401 hybrid_clock.cc:648] HybridClock initialized: now 1781958495725609 us; error 49 us; skew 500 ppm
Jun 20 12:28:15 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:15.729034 16401 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:28:15.729979 16401 webserver.cc:460] Webserver started at http://127.0.98.1:33719/ using document root <none> and password file <none>
I20260620 12:28:15.730827 16401 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:15.730939 16401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:15.735222 16401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:28:15.737528 16437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:15.738866 16401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:28:15.739001 16401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "6a7a764ab60844f9b4b011634b529269"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "daf4eabb031a6a6e70987c5f8973766d"
server_key_iv: "d332fb7d50269608d7caf038a0bf89db"
server_key_version: "encryptionkey@0"
I20260620 12:28:15.739567 16401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:15.764104 16401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:15.767578 16401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:15.767810 16401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:15.768497 16401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:15.769627 16444 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:28:15.774739 16401 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:28:15.774824 16401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 12:28:15.774911 16401 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:28:15.776618 16401 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:28:15.776737 16401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:15.777073 16444 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Bootstrap starting.
I20260620 12:28:15.787696 16401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46325
I20260620 12:28:15.787772 16551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46325 every 8 connection(s)
I20260620 12:28:15.788683 16401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:28:15.796237 16444 log.cc:824] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:15.797196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16401
I20260620 12:28:15.797688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42733
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46423
--webserver_port=34773
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:28:15 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:15.804634 16444 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:28:15.805081 16444 tablet_bootstrap.cc:492] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Bootstrap complete.
I20260620 12:28:15.806331 16444 ts_tablet_manager.cc:1397] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260620 12:28:15.807226 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:15.790836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55723 (local address 127.0.98.62:46423)
0620 12:28:15.791290 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:15.791312 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:15.791571 (+ 259us) server_negotiation.cc:407] Connection header received
0620 12:28:15.792731 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:15.792750 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:15.793166 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:15.793553 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:15.794589 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.796043 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.797304 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.798014 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.798192 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:15.801139 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:15.801145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:15.801159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:15.801190 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:15.803385 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:15.804052 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:15.804058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:15.804063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:15.804209 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:15.804653 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:15.804659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:15.804660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:15.805165 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:15.805298 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:15.806014 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 12:28:15.806313 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":135,"threads_started":1}
I20260620 12:28:15.807382 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:15.790780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:55723)
0620 12:28:15.791376 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:15.791429 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:15.792247 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:15.793600 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:15.793606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:15.793949 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:15.794411 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:15.794425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.796185 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.796188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:15.797166 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:15.797177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.798163 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.798167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:15.798342 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:15.799033 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:15.799054 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:15.800935 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:15.803514 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:15.803524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:15.803540 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:15.803919 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:15.804321 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:15.804327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:15.804331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:15.804517 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:15.805300 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:15.805309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:15.805654 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 12:28:15.805872 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 12:28:15.806228 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":128,"threads_started":1}
I20260620 12:28:15.809180 16444 raft_consensus.cc:348] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:28:15.809609 16444 raft_consensus.cc:729] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7a764ab60844f9b4b011634b529269, State: Initialized, Role: FOLLOWER
I20260620 12:28:15.810123 16444 consensus_queue.cc:260] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } } peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } }
I20260620 12:28:15.810348 16552 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46423
I20260620 12:28:15.810566 16552 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:15.811101 16552 heartbeater.cc:511] Master 127.0.98.62:46423 requested a full tablet report, sending...
I20260620 12:28:15.811136 16444 ts_tablet_manager.cc:1428] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:28:15.813504 16363 ts_manager.cc:194] Registered new tserver with Master: 6a7a764ab60844f9b4b011634b529269 (127.0.98.1:46325)
I20260620 12:28:15.817381 16363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55723
I20260620 12:28:15.819984 16552 heartbeater.cc:503] Master 127.0.98.62:46423 was elected leader, sending a full tablet report...
Jun 20 12:28:15 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:28:15.827776 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:15.816697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46325 (local address 127.0.0.1:34500)
0620 12:28:15.817162 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:15.817204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:15.817328 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:15.817834 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:15.817839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:15.818267 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:15.818366 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:15.818372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.819742 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.819746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:15.820772 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:15.820783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.821632 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.821644 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:15.821726 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:15.822306 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:15.822317 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:15.824021 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:15.825992 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:15.826002 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:15.826007 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:15.826234 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:15.826554 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:15.826558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:15.826560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:15.826697 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:15.827129 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:15.827135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:15.827261 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 12:28:15.827401 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 12:28:15.827593 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":87,"threads_started":1}
I20260620 12:28:15.827781 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:15.816737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.0.98.1:46325)
0620 12:28:15.817413 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:15.817424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:15.817445 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:28:15.817510 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:15.817517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:15.817690 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:15.817831 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:15.818555 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.819618 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.820916 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:15.821432 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:15.821532 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:15.824166 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:15.824175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:15.824187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:15.824217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:15.825868 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:15.826352 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:15.826357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:15.826359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:15.826442 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:15.826774 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:15.826779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:15.826780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:15.827011 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:15.827153 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:15.827489 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 12:28:15.827645 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":80,"threads_started":1}
I20260620 12:28:15.828713 16486 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e39be7df31842d2bb0f65b0f9b7d55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34500
I20260620 12:28:15.829572 16561 tablet_replica.cc:330] stopping tablet replica
I20260620 12:28:15.829870 16561 raft_consensus.cc:2229] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:15.830127 16561 raft_consensus.cc:2258] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:15.831205 16561 ts_tablet_manager.cc:1905] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:15.835570 16561 ts_tablet_manager.cc:1918] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:15.835665 16561 log.cc:1196] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal/wals/6e39be7df31842d2bb0f65b0f9b7d55b
I20260620 12:28:15.835934 16561 ts_tablet_manager.cc:1939] T 6e39be7df31842d2bb0f65b0f9b7d55b P 6a7a764ab60844f9b4b011634b529269: Deleting consensus metadata
I20260620 12:28:15.837718 16349 catalog_manager.cc:4841] TS 6a7a764ab60844f9b4b011634b529269 (127.0.98.1:46325): tablet 6e39be7df31842d2bb0f65b0f9b7d55b (table test-table [id=3a47237531f54a109ff2e6423df9885f]) successfully deleted
W20260620 12:28:15.914248 16557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:15.914486 16557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:15.914557 16557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:15.918514 16557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:15.918562 16557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:15.918663 16557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:28:15.924126 16557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34773
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:15.925256 16557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:15.926254 16557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:15.935598 16569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:15.935647 16566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:15.935598 16567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:15.935812 16557 server_base.cc:1034] running on GCE node
I20260620 12:28:15.936394 16557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:15.937098 16557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:15.938305 16557 hybrid_clock.cc:648] HybridClock initialized: now 1781958495938272 us; error 54 us; skew 500 ppm
Jun 20 12:28:15 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:15.941056 16557 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:28:15.941936 16557 webserver.cc:460] Webserver started at http://127.0.98.2:34773/ using document root <none> and password file <none>
I20260620 12:28:15.942718 16557 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:15.942853 16557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:15.948236 16557 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:28:15.950784 16576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:15.952343 16557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:28:15.952457 16557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "3d156f39eefa46dbac4936e1fbf64662"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "ece1a4a09166c5de8a151f69a73ef867"
server_key_iv: "0fcbf943f4fd2b0484f3ee2ba9700d4c"
server_key_version: "encryptionkey@0"
I20260620 12:28:15.953186 16557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:15.984483 16557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:15.988489 16557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:15.988704 16557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:15.989354 16557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:15.990384 16557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:15.990430 16557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:15.990499 16557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:15.990530 16557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:15.999904 16557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42733
I20260620 12:28:15.999933 16689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42733 every 8 connection(s)
I20260620 12:28:16.000941 16557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:28:16.004760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16557
I20260620 12:28:16.005282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40829
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46423
--webserver_port=42677
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:28:16 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:16.017278 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.002800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40211 (local address 127.0.98.62:46423)
0620 12:28:16.002967 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.002972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.004277 (+ 1305us) server_negotiation.cc:407] Connection header received
0620 12:28:16.005533 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.005539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.005606 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.005666 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:16.007441 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.008262 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.010151 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.010378 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.010468 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.013662 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:16.013665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:16.013667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:16.013694 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:16.015218 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.015741 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.015749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.015751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.015820 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.016176 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.016179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.016181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.016358 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:16.016464 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.016961 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.017101 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:28:16.018041 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.003226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:40211)
0620 12:28:16.004113 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:16.004156 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:16.005256 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:16.006094 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:16.006105 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:16.006673 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:16.007251 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.007265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.008412 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.008418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.010013 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.010028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.010620 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.010629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.010969 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.011644 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:16.011676 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:16.013486 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:16.015347 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:16.015353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:16.015367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:16.015630 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:16.015929 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:16.015932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:16.015934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:16.016074 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:16.016463 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:16.016469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:16.016716 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 12:28:16.016901 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 12:28:16.017231 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":532,"threads_started":1}
I20260620 12:28:16.019245 16690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46423
I20260620 12:28:16.019518 16690 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:16.020222 16690 heartbeater.cc:511] Master 127.0.98.62:46423 requested a full tablet report, sending...
I20260620 12:28:16.021289 16363 ts_manager.cc:194] Registered new tserver with Master: 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733)
I20260620 12:28:16.021898 16363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40211
W20260620 12:28:16.117734 16693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:16.117980 16693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:16.118055 16693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:16.122130 16693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:16.122208 16693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:16.122299 16693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:28:16.127897 16693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42677
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:16.128904 16693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:16.129896 16693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:16.139328 16701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:16.139298 16698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:16.139441 16699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:16.139679 16693 server_base.cc:1034] running on GCE node
I20260620 12:28:16.140076 16693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:16.140666 16693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:16.141827 16693 hybrid_clock.cc:648] HybridClock initialized: now 1781958496141802 us; error 37 us; skew 500 ppm
Jun 20 12:28:16 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:16.145507 16693 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:28:16.146384 16693 webserver.cc:460] Webserver started at http://127.0.98.3:42677/ using document root <none> and password file <none>
I20260620 12:28:16.147007 16693 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:16.147107 16693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:16.152194 16693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 12:28:16.154356 16708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:16.155640 16693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:16.155758 16693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "efc0c5775e8e41f2995f16ed8c6e83a5"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "195356b887faee835a61aa249c39acf5"
server_key_iv: "5ed3034f196388ac4b3e88863379869e"
server_key_version: "encryptionkey@0"
I20260620 12:28:16.156272 16693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:16.180258 16693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:16.183207 16693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:16.183413 16693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:16.184007 16693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:16.184849 16693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:16.184893 16693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:16.184964 16693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:16.184998 16693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:16.194123 16693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40829
I20260620 12:28:16.194196 16821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40829 every 8 connection(s)
I20260620 12:28:16.195135 16693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:28:16.202368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16693
Jun 20 12:28:16 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:16.209399 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.196953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43995 (local address 127.0.98.62:46423)
0620 12:28:16.197085 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.197090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.197891 (+ 801us) server_negotiation.cc:407] Connection header received
0620 12:28:16.199230 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.199233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.199312 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.199389 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:16.200488 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.201013 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.202144 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.202527 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.202637 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.205325 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:16.205329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:16.205331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:16.205363 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:16.206899 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.207538 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.207541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.207542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.207619 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.208050 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.208053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.208055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.208303 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:16.208399 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.209000 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.209153 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:28:16.210258 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.197325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:43995)
0620 12:28:16.197736 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:16.197777 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:16.199027 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:16.199551 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:16.199557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:16.199892 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:16.200352 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.200365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.201119 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.201121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.201994 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.202003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.202701 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.202705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.202907 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.203570 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:16.203585 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:16.205151 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:16.207036 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:16.207043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:16.207059 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:16.207418 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:16.207722 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:16.207726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:16.207730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:16.207910 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:16.208422 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:16.208429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:16.208729 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 12:28:16.208927 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 12:28:16.209231 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260620 12:28:16.211443 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.203031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.0.98.62:46423)
0620 12:28:16.203600 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.203604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.203623 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:28:16.203651 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.203653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.203685 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.203790 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:16.204664 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.205255 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.206118 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.206345 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.206449 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.208066 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:16.208069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:16.208072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:16.208093 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:16.209721 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.210187 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.210189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.210190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.210259 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.210559 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.210562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.210564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.210787 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:16.211011 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.211163 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.211285 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":93,"threads_started":1}
I20260620 12:28:16.211467 16822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46423
I20260620 12:28:16.211704 16822 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:16.212255 16822 heartbeater.cc:511] Master 127.0.98.62:46423 requested a full tablet report, sending...
I20260620 12:28:16.213258 16363 ts_manager.cc:194] Registered new tserver with Master: efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
I20260620 12:28:16.213745 16363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43995
I20260620 12:28:16.225028 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:28:16.229008 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.225684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.0.98.62:46423)
0620 12:28:16.225830 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.225834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.225911 (+ 77us) server_negotiation.cc:407] Connection header received
0620 12:28:16.226067 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.226069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.226161 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.226246 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:28:16.226483 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.227170 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.228015 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.228225 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.228300 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.228585 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:28:16.228715 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 12:28:16.228909 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53664: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:28:16.229096 16825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53664: token expired
W20260620 12:28:16.229437 16291 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46423)
I20260620 12:28:16.236166 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.229755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.0.98.62:46423)
0620 12:28:16.229859 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.229863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.229882 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:28:16.229943 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.229945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.229986 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.230046 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:16.230599 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.231121 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.231864 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.232055 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.232129 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.233419 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:16.233421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:16.233423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:16.233443 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:16.234772 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.235170 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.235173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.235175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.235252 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:16.235532 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:16.235535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:16.235536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:16.235693 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:16.235797 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.235979 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.236061 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:28:16.238616 16363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:28:16.241806 16363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:28:16.249589 16486 tablet_service.cc:1467] Processing CreateTablet for tablet 62ef4b051c3446c5bf339b87ce62eda1 (DEFAULT_TABLE table=test-table [id=617c1118cba14a2ab41ee8bf4148593b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:16.250274 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ef4b051c3446c5bf339b87ce62eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:16.252945 16444 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Bootstrap starting.
I20260620 12:28:16.253827 16444 tablet_bootstrap.cc:654] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:16.254602 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.249221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.0.98.3:40829)
0620 12:28:16.249538 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.249548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.249567 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:28:16.249629 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.249633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.249799 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.249948 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:16.250250 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.251443 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.253076 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.253901 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.254051 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.254190 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.254402 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.254484 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":108,"threads_started":1}
I20260620 12:28:16.254613 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.248802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40829 (local address 127.0.0.1:48492)
0620 12:28:16.249032 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:16.249045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:16.249121 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:16.249939 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:16.249942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:16.249962 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:16.250111 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.250117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.251560 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.251563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.252922 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.252934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.254087 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.254090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.254210 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.254271 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 12:28:16.254368 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 12:28:16.254438 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":66,"thread_start_us":113,"threads_started":1}
I20260620 12:28:16.255318 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.248601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42733 (local address 127.0.0.1:55826)
0620 12:28:16.248762 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:16.248775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:16.248861 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:16.249848 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:16.249853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:16.249875 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:16.250161 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.250172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.251428 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.251433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.252923 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.252934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.254904 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.254910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.255019 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.255043 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:28:16.255120 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:28:16.255202 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":263}
I20260620 12:28:16.255508 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.248775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.0.98.2:42733)
0620 12:28:16.249315 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.249328 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.249368 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:28:16.249453 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.249459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.249676 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.250335 (+ 659us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:16.250503 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.251311 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.253076 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.253901 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.254051 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.254190 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.255264 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.255365 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":134,"threads_started":1}
I20260620 12:28:16.256142 16444 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: No bootstrap required, opened a new log
I20260620 12:28:16.256238 16444 ts_tablet_manager.cc:1397] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:28:16.256825 16444 raft_consensus.cc:348] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.256953 16444 raft_consensus.cc:374] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:16.257001 16444 raft_consensus.cc:729] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7a764ab60844f9b4b011634b529269, State: Initialized, Role: FOLLOWER
I20260620 12:28:16.257130 16444 consensus_queue.cc:260] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.257422 16444 ts_tablet_manager.cc:1428] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:28:16.257913 16756 tablet_service.cc:1467] Processing CreateTablet for tablet 62ef4b051c3446c5bf339b87ce62eda1 (DEFAULT_TABLE table=test-table [id=617c1118cba14a2ab41ee8bf4148593b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:16.258457 16624 tablet_service.cc:1467] Processing CreateTablet for tablet 62ef4b051c3446c5bf339b87ce62eda1 (DEFAULT_TABLE table=test-table [id=617c1118cba14a2ab41ee8bf4148593b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:16.259073 16756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ef4b051c3446c5bf339b87ce62eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:16.259500 16624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ef4b051c3446c5bf339b87ce62eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:16.267127 16835 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Bootstrap starting.
I20260620 12:28:16.267127 16836 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Bootstrap starting.
I20260620 12:28:16.269459 16836 tablet_bootstrap.cc:654] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:16.269897 16835 tablet_bootstrap.cc:654] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:16.270821 16835 log.cc:824] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:16.270821 16836 log.cc:824] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:16.272686 16835 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: No bootstrap required, opened a new log
I20260620 12:28:16.272693 16836 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: No bootstrap required, opened a new log
I20260620 12:28:16.272907 16835 ts_tablet_manager.cc:1397] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:28:16.272921 16836 ts_tablet_manager.cc:1397] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 12:28:16.275750 16835 raft_consensus.cc:348] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.275750 16836 raft_consensus.cc:348] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.275990 16836 raft_consensus.cc:374] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:16.275996 16835 raft_consensus.cc:374] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:16.276077 16836 raft_consensus.cc:729] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc0c5775e8e41f2995f16ed8c6e83a5, State: Initialized, Role: FOLLOWER
I20260620 12:28:16.276103 16835 raft_consensus.cc:729] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d156f39eefa46dbac4936e1fbf64662, State: Initialized, Role: FOLLOWER
I20260620 12:28:16.276513 16836 consensus_queue.cc:260] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.276515 16835 consensus_queue.cc:260] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.277117 16822 heartbeater.cc:503] Master 127.0.98.62:46423 was elected leader, sending a full tablet report...
I20260620 12:28:16.277117 16690 heartbeater.cc:503] Master 127.0.98.62:46423 was elected leader, sending a full tablet report...
I20260620 12:28:16.277477 16836 ts_tablet_manager.cc:1428] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 12:28:16.277639 16835 ts_tablet_manager.cc:1428] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 12:28:16.289873 16553 tablet.cc:2369] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:28:16.382284 16558 raft_consensus.cc:482] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:16.382516 16558 raft_consensus.cc:504] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.383710 16558 leader_election.cc:288] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
I20260620 12:28:16.389174 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.384051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40365 (local address 127.0.98.2:42733)
0620 12:28:16.384190 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.384198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.384460 (+ 262us) server_negotiation.cc:407] Connection header received
0620 12:28:16.384670 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.384673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.384729 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.384821 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:16.385122 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.385914 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.387796 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.388466 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.388565 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.388627 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.388963 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.389046 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:28:16.389174 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.384229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41971 (local address 127.0.98.3:40829)
0620 12:28:16.384365 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.384373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.384526 (+ 153us) server_negotiation.cc:407] Connection header received
0620 12:28:16.384670 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.384673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.384728 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.384837 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:16.385120 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.385915 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.387804 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.388452 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.388565 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.388627 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.388963 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.389046 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:28:16.389228 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.384023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40829 (local address 127.0.98.1:41971)
0620 12:28:16.384455 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:16.384468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:16.384547 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:16.384850 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:16.384854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:16.384874 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:16.384994 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.385000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.386041 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.386044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.387645 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.387650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.388588 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.388592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.388739 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.388823 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 12:28:16.388902 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:28:16.389053 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":237,"thread_start_us":81,"threads_started":1}
I20260620 12:28:16.389250 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.383898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42733 (local address 127.0.98.1:40365)
0620 12:28:16.384333 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:16.384349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:16.384495 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:16.384850 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:16.384854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:16.384924 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:16.384994 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.385000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.386041 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.386044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.387635 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:16.387649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.388588 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.388592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:16.388742 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.388818 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 12:28:16.388902 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 12:28:16.389053 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":276,"spinlock_wait_cycles":36864,"thread_start_us":118,"threads_started":1}
I20260620 12:28:16.389873 16644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ef4b051c3446c5bf339b87ce62eda1" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d156f39eefa46dbac4936e1fbf64662" is_pre_election: true
I20260620 12:28:16.390221 16644 raft_consensus.cc:2454] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 0.
I20260620 12:28:16.390265 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ef4b051c3446c5bf339b87ce62eda1" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" is_pre_election: true
I20260620 12:28:16.390544 16776 raft_consensus.cc:2454] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 0.
I20260620 12:28:16.390806 16440 leader_election.cc:302] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d156f39eefa46dbac4936e1fbf64662, 6a7a764ab60844f9b4b011634b529269; no voters:
I20260620 12:28:16.391117 16558 raft_consensus.cc:2781] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:28:16.391223 16558 raft_consensus.cc:482] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:28:16.391309 16558 raft_consensus.cc:3037] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:16.392380 16558 raft_consensus.cc:504] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.392733 16558 leader_election.cc:288] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 election: Requested vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
I20260620 12:28:16.393115 16644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ef4b051c3446c5bf339b87ce62eda1" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d156f39eefa46dbac4936e1fbf64662"
I20260620 12:28:16.393143 16776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ef4b051c3446c5bf339b87ce62eda1" candidate_uuid: "6a7a764ab60844f9b4b011634b529269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5"
I20260620 12:28:16.393244 16776 raft_consensus.cc:3037] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:16.393288 16644 raft_consensus.cc:3037] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:16.394565 16776 raft_consensus.cc:2454] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 1.
I20260620 12:28:16.394564 16644 raft_consensus.cc:2454] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7a764ab60844f9b4b011634b529269 in term 1.
I20260620 12:28:16.394930 16438 leader_election.cc:302] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d156f39eefa46dbac4936e1fbf64662, 6a7a764ab60844f9b4b011634b529269, efc0c5775e8e41f2995f16ed8c6e83a5; no voters:
I20260620 12:28:16.395171 16558 raft_consensus.cc:2781] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:16.395326 16558 raft_consensus.cc:686] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7a764ab60844f9b4b011634b529269, State: Running, Role: LEADER
I20260620 12:28:16.395589 16558 consensus_queue.cc:237] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:16.398576 16363 catalog_manager.cc:5495] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7a764ab60844f9b4b011634b529269 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6a7a764ab60844f9b4b011634b529269" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } health_report { overall_health: HEALTHY } } }
I20260620 12:28:16.442744 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.439173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.98.1:46325)
0620 12:28:16.439477 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:16.439484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:16.439498 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:28:16.439570 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:16.439572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:16.439628 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:16.439734 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:28:16.439936 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.440468 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.441223 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:16.441468 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:16.441547 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:16.441769 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:28:16.442185 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:28:16.442406 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:16.442500 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 12:28:16.442553 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":147,"threads_started":1}
W20260620 12:28:16.446173 16823 tablet.cc:2369] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:28:16.446822 16776 raft_consensus.cc:1261] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Refusing update from remote peer 6a7a764ab60844f9b4b011634b529269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:16.446825 16644 raft_consensus.cc:1261] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Refusing update from remote peer 6a7a764ab60844f9b4b011634b529269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:16.447486 16558 consensus_queue.cc:1035] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:16.447708 16843 consensus_queue.cc:1035] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:16.455524 16848 mvcc.cc:205] Tried to move back new op lower bound from 7298902001439137792 to 7298902001238712320. Current Snapshot: MvccSnapshot[applied={T|T < 7298902001439137792}]
I20260620 12:28:16.456663 16847 mvcc.cc:205] Tried to move back new op lower bound from 7298902001439137792 to 7298902001238712320. Current Snapshot: MvccSnapshot[applied={T|T < 7298902001439137792}]
I20260620 12:28:16.462888 16363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:16.463017 16363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:16.465193 16363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Sending DeleteTablet for 3 replicas of tablet 62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:16.465776 16624 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ef4b051c3446c5bf339b87ce62eda1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55826
I20260620 12:28:16.465857 16756 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ef4b051c3446c5bf339b87ce62eda1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48492
I20260620 12:28:16.466010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 16401
I20260620 12:28:16.466847 16857 tablet_replica.cc:330] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: stopping tablet replica
I20260620 12:28:16.466886 16856 tablet_replica.cc:330] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: stopping tablet replica
I20260620 12:28:16.467193 16857 raft_consensus.cc:2229] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:16.467247 16856 raft_consensus.cc:2229] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:16.467559 16857 raft_consensus.cc:2258] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:16.467569 16856 raft_consensus.cc:2258] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:16.468842 16857 ts_tablet_manager.cc:1905] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:16.468919 16856 ts_tablet_manager.cc:1905] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:16.471001 16857 ts_tablet_manager.cc:1918] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:16.471112 16857 log.cc:1196] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:16.471614 16857 ts_tablet_manager.cc:1939] T 62ef4b051c3446c5bf339b87ce62eda1 P efc0c5775e8e41f2995f16ed8c6e83a5: Deleting consensus metadata
I20260620 12:28:16.472344 16856 ts_tablet_manager.cc:1918] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:16.472440 16856 log.cc:1196] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:16.472675 16347 catalog_manager.cc:4841] TS efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829): tablet 62ef4b051c3446c5bf339b87ce62eda1 (table test-table [id=617c1118cba14a2ab41ee8bf4148593b]) successfully deleted
I20260620 12:28:16.472797 16856 ts_tablet_manager.cc:1939] T 62ef4b051c3446c5bf339b87ce62eda1 P 3d156f39eefa46dbac4936e1fbf64662: Deleting consensus metadata
I20260620 12:28:16.473917 16349 catalog_manager.cc:4841] TS 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733): tablet 62ef4b051c3446c5bf339b87ce62eda1 (table test-table [id=617c1118cba14a2ab41ee8bf4148593b]) successfully deleted
W20260620 12:28:16.474278 16349 connection.cc:535] client connection to 127.0.98.1:46325 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 12:28:16.474431 16349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:28:16.474786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 16557
I20260620 12:28:16.476976 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:16.476710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46325 (local address 127.0.0.1:60438)
0620 12:28:16.476836 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:16.476912 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 12:28:16.477208 16349 catalog_manager.cc:4584] TS 6a7a764ab60844f9b4b011634b529269 (127.0.98.1:46325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62ef4b051c3446c5bf339b87ce62eda1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46325: connect: Connection refused (error 111)
I20260620 12:28:16.481271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 16693
I20260620 12:28:16.486630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 16330
I20260620 12:28:36.493017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--webserver_interface=127.0.98.62
--webserver_port=39893
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:36.601380 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:36.601619 16859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:36.601657 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:36.605489 16859 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:28:36.605531 16859 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:28:36.605546 16859 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:28:36.605561 16859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:28:36.605577 16859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:36.605594 16859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:36.605610 16859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:28:36.605625 16859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:28:36.610610 16859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:36.611562 16859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:36.612476 16859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:28:36.621292 16859 server_base.cc:1034] running on GCE node
W20260620 12:28:36.621236 16864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:36.621212 16865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:36.621253 16867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:36.622013 16859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:36.623131 16859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:36.624322 16859 hybrid_clock.cc:648] HybridClock initialized: now 1781958516624293 us; error 40 us; skew 500 ppm
Jun 20 12:28:36 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:36.627219 16859 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:28:36.628079 16859 webserver.cc:460] Webserver started at http://127.0.98.62:39893/ using document root <none> and password file <none>
I20260620 12:28:36.628644 16859 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:36.628736 16859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:36.632494 16859 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 12:28:36.634501 16874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:36.635720 16859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:36.635854 16859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "a9e72ea8ce60416db6469e299e29bbdd"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "07c999ac196bff5f713ca287c32d1f4b"
server_key_iv: "ba5176c62ac444ab0bfc4758a643af6b"
server_key_version: "encryptionkey@0"
I20260620 12:28:36.636524 16859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:36.659715 16859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:36.662441 16859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:36.662643 16859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:36.669477 16859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46423
I20260620 12:28:36.669767 16926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46423 every 8 connection(s)
I20260620 12:28:36.670524 16859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:28:36.675074 16927 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:28:36.677183 16927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Bootstrap starting.
I20260620 12:28:36.679291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16859
I20260620 12:28:36.679795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46325
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46423
--webserver_port=33719
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:28:36.695377 16927 log.cc:824] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:36.707484 16927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:28:36.707894 16927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Bootstrap complete.
I20260620 12:28:36.712000 16927 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:36.712378 16927 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e72ea8ce60416db6469e299e29bbdd, State: Initialized, Role: FOLLOWER
I20260620 12:28:36.712875 16927 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:36.712988 16927 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:28:36.713073 16927 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:28:36.713186 16927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:28:36.715166 16927 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:36.715497 16927 leader_election.cc:302] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e72ea8ce60416db6469e299e29bbdd; no voters:
I20260620 12:28:36.715874 16927 leader_election.cc:288] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 12:28:36.715929 16931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:28:36.716117 16931 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [term 3 LEADER]: Becoming Leader. State: Replica: a9e72ea8ce60416db6469e299e29bbdd, State: Running, Role: LEADER
I20260620 12:28:36.716481 16931 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260620 12:28:36.717073 16927 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:28:36.717763 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e72ea8ce60416db6469e299e29bbdd. Latest consensus state: current_term: 3 leader_uuid: "a9e72ea8ce60416db6469e299e29bbdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260620 12:28:36.718184 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:36.718086 16933 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a9e72ea8ce60416db6469e299e29bbdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e72ea8ce60416db6469e299e29bbdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260620 12:28:36.718534 16933 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:36.718907 16941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:28:36.724067 16941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:28:36.724885 16941 catalog_manager.cc:1259] Loaded cluster ID: 8d9747ac42a94b2b804fd3e6289d4c0c
I20260620 12:28:36.725021 16941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:28:36.727717 16941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:28:36.728437 16941 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Loaded TSK: 0
I20260620 12:28:36.728519 16941 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9e72ea8ce60416db6469e299e29bbdd: Loaded TSK: 1
F20260620 12:28:36.729249 16941 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4e13fb99ad google::LogMessage::Fail() at ??:0
@ 0x7f4e13fbc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4e13fb94de google::LogMessage::Flush() at ??:0
@ 0x7f4e13fbaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4e16c7d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4e16c75f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4e1e91e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4e1e8f899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4e1e8f9178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4e1e93adf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4e1e95982b std::function<>::operator()() at ??:0
@ 0x7f4e1e8f8b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4e1e8f9f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4e1e8f6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4e1e93a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4e1620b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e1620bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4e1620d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4e161fc512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e178406db start_thread at ??:0
@ 0x7f4e12ab761f clone at ??:0
W20260620 12:28:36.835664 16929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:36.835934 16929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:36.836023 16929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:36.840185 16929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:36.840245 16929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:36.840335 16929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:28:36.845649 16929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33719
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:36.846824 16929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:36.847880 16929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:36.857640 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:36.857631 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:36.857899 16929 server_base.cc:1034] running on GCE node
W20260620 12:28:36.857905 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:36.858361 16929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:36.859107 16929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:36.860299 16929 hybrid_clock.cc:648] HybridClock initialized: now 1781958516860276 us; error 47 us; skew 500 ppm
Jun 20 12:28:36 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:36.863158 16929 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:28:36.864053 16929 webserver.cc:460] Webserver started at http://127.0.98.1:33719/ using document root <none> and password file <none>
I20260620 12:28:36.864645 16929 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:36.864738 16929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:36.868835 16929 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:28:36.871248 16984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:36.872579 16929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:28:36.872718 16929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "6a7a764ab60844f9b4b011634b529269"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "daf4eabb031a6a6e70987c5f8973766d"
server_key_iv: "d332fb7d50269608d7caf038a0bf89db"
server_key_version: "encryptionkey@0"
I20260620 12:28:36.873281 16929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:36.888082 16929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:36.891395 16929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:36.891605 16929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:36.892189 16929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:36.893388 16991 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:28:36.898700 16929 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:28:36.898769 16929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 12:28:36.898849 16929 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:28:36.900482 16929 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:28:36.900553 16929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 12:28:36.900856 16991 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Bootstrap starting.
I20260620 12:28:36.911434 16929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46325
I20260620 12:28:36.911657 17098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46325 every 8 connection(s)
I20260620 12:28:36.912438 16929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:28:36.917039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 16929
I20260620 12:28:36.917483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42733
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46423
--webserver_port=34773
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:28:36.922928 16991 log.cc:824] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:36.931955 16991 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:28:36.932384 16991 tablet_bootstrap.cc:492] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Bootstrap complete.
I20260620 12:28:36.933815 16991 ts_tablet_manager.cc:1397] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.008s
I20260620 12:28:36.937139 16991 raft_consensus.cc:348] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:36.937709 16991 raft_consensus.cc:729] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7a764ab60844f9b4b011634b529269, State: Initialized, Role: FOLLOWER
I20260620 12:28:36.938246 16991 consensus_queue.cc:260] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:36.939313 16991 ts_tablet_manager.cc:1428] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260620 12:28:36.963256 17101 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:28:36.914372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:40933)
0620 12:28:36.914877 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:36.914929 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:36.915739 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:36.962395 (+ 46656us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":137,"threads_started":1}
W20260620 12:28:36.963804 17099 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46423 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46423: Client connection negotiation failed: client connection to 127.0.98.62:46423: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:28:36.964648 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:36.964366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:58689)
0620 12:28:36.964523 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:36.964591 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 12:28:36.965451 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:36.965233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:36637)
0620 12:28:36.965358 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:36.965405 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 12:28:36.965710 17099 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:28:37.034508 17102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:37.034807 17102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:37.034898 17102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:37.038908 17102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:37.038971 17102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:37.039080 17102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:28:37.044303 17102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34773
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:37.045357 17102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:37.046393 17102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:37.056447 17109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:37.056447 17110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:37.056458 17112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:37.056689 17102 server_base.cc:1034] running on GCE node
I20260620 12:28:37.057163 17102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:37.057832 17102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:37.059041 17102 hybrid_clock.cc:648] HybridClock initialized: now 1781958517059004 us; error 47 us; skew 500 ppm
Jun 20 12:28:37 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:37.061787 17102 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:28:37.062605 17102 webserver.cc:460] Webserver started at http://127.0.98.2:34773/ using document root <none> and password file <none>
I20260620 12:28:37.063176 17102 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:37.063239 17102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:37.067350 17102 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:28:37.069491 17119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:37.070873 17102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:28:37.071003 17102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "3d156f39eefa46dbac4936e1fbf64662"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "ece1a4a09166c5de8a151f69a73ef867"
server_key_iv: "0fcbf943f4fd2b0484f3ee2ba9700d4c"
server_key_version: "encryptionkey@0"
I20260620 12:28:37.071607 17102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:37.097054 17102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:37.100168 17102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:37.100394 17102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:37.101027 17102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:37.101899 17102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:37.101950 17102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:37.101992 17102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:37.102057 17102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:37.112385 17102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42733
I20260620 12:28:37.112422 17232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42733 every 8 connection(s)
I20260620 12:28:37.113443 17102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:28:37.114534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 17102
I20260620 12:28:37.115069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40829
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46423
--webserver_port=42677
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:28:37.118315 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:37.116027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:42417)
0620 12:28:37.116678 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:37.117422 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":170,"threads_started":1}
W20260620 12:28:37.118818 17233 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46423 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46423: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
I20260620 12:28:37.119454 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:37.119200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:58999)
0620 12:28:37.119324 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:37.119385 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 12:28:37.120142 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:37.119928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:42689)
0620 12:28:37.120032 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:37.120085 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 12:28:37.120301 17233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:28:37.223485 17236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:37.223776 17236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:37.223863 17236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:37.228103 17236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:37.228191 17236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:37.228298 17236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:28:37.233332 17236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42677
--tserver_master_addrs=127.0.98.62:46423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:37.234333 17236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:37.235327 17236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:37.244784 17242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:37.244797 17241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:37.244832 17236 server_base.cc:1034] running on GCE node
W20260620 12:28:37.245035 17244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:37.245424 17236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:37.246068 17236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:37.247252 17236 hybrid_clock.cc:648] HybridClock initialized: now 1781958517247240 us; error 23 us; skew 500 ppm
Jun 20 12:28:37 dist-test-slave-bwpp krb5kdc[15782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:37.249784 17236 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:28:37.250545 17236 webserver.cc:460] Webserver started at http://127.0.98.3:42677/ using document root <none> and password file <none>
I20260620 12:28:37.251088 17236 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:37.251149 17236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:37.254804 17236 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 12:28:37.256661 17251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:37.257846 17236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:37.257961 17236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "efc0c5775e8e41f2995f16ed8c6e83a5"
format_stamp: "Formatted at 2026-06-20 12:27:54 on dist-test-slave-bwpp"
server_key: "195356b887faee835a61aa249c39acf5"
server_key_iv: "5ed3034f196388ac4b3e88863379869e"
server_key_version: "encryptionkey@0"
I20260620 12:28:37.258457 17236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:37.282465 17236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:37.285180 17236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:37.285370 17236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:37.285930 17236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:37.286785 17236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:37.286832 17236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:37.286895 17236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:37.286931 17236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:37.295830 17236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40829
I20260620 12:28:37.295850 17364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40829 every 8 connection(s)
I20260620 12:28:37.296813 17236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:28:37.300539 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:37.298913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:40845)
0620 12:28:37.299359 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:37.299847 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
W20260620 12:28:37.300995 17365 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46423 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46423: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
I20260620 12:28:37.301694 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:37.301351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:34655)
0620 12:28:37.301450 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:37.301642 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 12:28:37.302352 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:37.302140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:57607)
0620 12:28:37.302233 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:37.302309 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 12:28:37.302546 17365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:28:37.302759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 17236
I20260620 12:28:37.967005 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:37.966364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:34675)
0620 12:28:37.966635 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:37.966886 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260620 12:28:38.121214 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:38.120804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:43921)
0620 12:28:38.121062 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:38.121116 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":66,"threads_started":1}
I20260620 12:28:38.303493 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:38.303059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:41207)
0620 12:28:38.303330 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:38.303394 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260620 12:28:38.662731 17372 raft_consensus.cc:482] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:38.662959 17372 raft_consensus.cc:504] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:38.664064 17372 leader_election.cc:288] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
Jun 20 12:28:38 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 12:28:38 dist-test-slave-bwpp krb5kdc[15782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 12:28:38.681536 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:38.664272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42733 (local address 127.0.98.1:56713)
0620 12:28:38.664693 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:38.664707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:38.664831 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:38.666509 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:38.666512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:38.667076 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:38.667538 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:38.667552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.668811 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.668816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:38.669714 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:38.669723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.670458 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.670462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:38.670715 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:38.674343 (+ 3628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:38.674356 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:38.676180 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:38.678787 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:38.678792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:38.678805 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:38.679084 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:38.679452 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:38.679456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:38.679458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:38.679652 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:38.680414 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:38.680419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:38.680715 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 12:28:38.680946 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 12:28:38.681318 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":157,"spinlock_wait_cycles":22784,"thread_start_us":60,"threads_started":1}
I20260620 12:28:38.681560 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:38.664481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56713 (local address 127.0.98.2:42733)
0620 12:28:38.665101 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:38.665112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:38.665145 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:28:38.665777 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:38.665783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:38.666258 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:38.666572 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:38.667706 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.668669 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.669841 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.670312 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.670485 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:38.676496 (+ 6011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:38.676502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:38.676515 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:38.676543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:38.678642 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:38.679234 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:38.679239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:38.679243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:38.679352 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:38.679746 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:38.679749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:38.679751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:38.680298 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:38.680432 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:38.681083 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 12:28:38.681372 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":72,"threads_started":1}
I20260620 12:28:38.681560 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:38.664758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53109 (local address 127.0.98.3:40829)
0620 12:28:38.665068 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:38.665088 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:38.665123 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:28:38.665779 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:38.665785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:38.666120 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:38.666461 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:38.667714 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.668763 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.669842 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.670317 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.670497 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:38.676736 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:38.676746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:38.676759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:38.676788 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:38.679054 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:38.679597 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:38.679602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:38.679605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:38.679737 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:38.680011 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:38.680014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:38.680016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:38.680467 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:38.680570 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:38.681099 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 12:28:38.681372 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
I20260620 12:28:38.683682 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:38.664272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40829 (local address 127.0.98.1:53109)
0620 12:28:38.664639 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:38.664655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:38.664831 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:38.666492 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:38.666499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:38.667074 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:38.667539 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:38.667554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.668899 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.668904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:38.669718 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:38.669727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:38.670447 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:38.670450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:38.670671 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:38.674332 (+ 3661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:38.674348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:38.676547 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:38.679164 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:38.679171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:38.679176 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:38.679481 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:38.679833 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:38.679836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:38.679837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:38.679918 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:38.680572 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:38.680576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:38.680715 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 12:28:38.680946 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 12:28:38.683502 (+ 2556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":87,"spinlock_wait_cycles":134912,"thread_start_us":72,"threads_started":1}
W20260620 12:28:38.686064 16985 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
W20260620 12:28:38.686204 16987 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:38.686319 16987 leader_election.cc:302] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a7a764ab60844f9b4b011634b529269; no voters: 3d156f39eefa46dbac4936e1fbf64662, efc0c5775e8e41f2995f16ed8c6e83a5
I20260620 12:28:38.686587 17372 raft_consensus.cc:2726] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:28:38.968080 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:38.967788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:52369)
0620 12:28:38.967940 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:38.967996 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 12:28:39.122478 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:39.122076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:36045)
0620 12:28:39.122318 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:39.122371 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260620 12:28:39.304706 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:39.304312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:38271)
0620 12:28:39.304564 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:39.304624 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260620 12:28:39.969420 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:39.968943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:52705)
0620 12:28:39.969220 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:39.969294 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260620 12:28:40.123907 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:40.123382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:49573)
0620 12:28:40.123674 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:40.123740 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":113,"threads_started":1}
I20260620 12:28:40.305912 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:40.305525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:41983)
0620 12:28:40.305761 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:40.305813 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260620 12:28:40.745888 17382 raft_consensus.cc:482] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:40.746053 17382 raft_consensus.cc:504] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:40.746469 17382 leader_election.cc:288] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
W20260620 12:28:40.747332 16985 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
W20260620 12:28:40.747417 16987 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:40.747522 16987 leader_election.cc:302] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a7a764ab60844f9b4b011634b529269; no voters: 3d156f39eefa46dbac4936e1fbf64662, efc0c5775e8e41f2995f16ed8c6e83a5
I20260620 12:28:40.747699 17382 raft_consensus.cc:2726] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:28:40.970613 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:40.970204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:40661)
0620 12:28:40.970456 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:40.970514 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260620 12:28:41.124982 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:41.124602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:51955)
0620 12:28:41.124836 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:41.124893 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260620 12:28:41.307149 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:41.306754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:34657)
0620 12:28:41.307007 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:41.307061 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260620 12:28:41.972056 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:41.971572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:57477)
0620 12:28:41.971894 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:41.971957 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":72,"threads_started":1}
I20260620 12:28:42.126405 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:42.125928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:44893)
0620 12:28:42.126200 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:42.126304 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260620 12:28:42.248785 17388 raft_consensus.cc:482] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:42.248912 17388 raft_consensus.cc:504] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:42.249346 17388 leader_election.cc:288] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
W20260620 12:28:42.250172 16987 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
W20260620 12:28:42.250277 16985 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:42.250341 16985 leader_election.cc:302] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a7a764ab60844f9b4b011634b529269; no voters: 3d156f39eefa46dbac4936e1fbf64662, efc0c5775e8e41f2995f16ed8c6e83a5
I20260620 12:28:42.250486 17388 raft_consensus.cc:2726] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:28:42.308215 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:42.307842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:52261)
0620 12:28:42.308074 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:42.308128 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260620 12:28:42.973263 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:42.972837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:37335)
0620 12:28:42.973105 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:42.973167 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260620 12:28:43.127538 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:43.127133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:34737)
0620 12:28:43.127390 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:43.127447 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260620 12:28:43.309451 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:43.309017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:34423)
0620 12:28:43.309291 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:43.309357 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260620 12:28:43.974517 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:43.974058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:47645)
0620 12:28:43.974362 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:43.974421 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260620 12:28:44.128957 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:44.128363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:58975)
0620 12:28:44.128733 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:44.128810 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":84,"threads_started":1}
I20260620 12:28:44.310602 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:44.310168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:35635)
0620 12:28:44.310459 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:44.310510 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":71,"threads_started":1}
I20260620 12:28:44.975749 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:44.975324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:60981)
0620 12:28:44.975588 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:44.975653 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260620 12:28:45.130194 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:45.129752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:37479)
0620 12:28:45.130043 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:45.130106 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":71,"threads_started":1}
I20260620 12:28:45.311865 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:45.311425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:49163)
0620 12:28:45.311707 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:45.311772 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20260620 12:28:45.977111 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:45.976646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:38545)
0620 12:28:45.976951 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:45.977015 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":113,"threads_started":1}
I20260620 12:28:46.131383 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:46.131012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:42429)
0620 12:28:46.131235 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:46.131291 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260620 12:28:46.313035 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:46.312621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:47299)
0620 12:28:46.312876 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:46.312924 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260620 12:28:46.978300 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:46.977911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:53277)
0620 12:28:46.978147 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:46.978203 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260620 12:28:47.009840 17403 raft_consensus.cc:482] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:47.009985 17403 raft_consensus.cc:504] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:47.010460 17403 leader_election.cc:288] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
W20260620 12:28:47.011193 16985 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
W20260620 12:28:47.011283 16987 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:47.011391 16987 leader_election.cc:302] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a7a764ab60844f9b4b011634b529269; no voters: 3d156f39eefa46dbac4936e1fbf64662, efc0c5775e8e41f2995f16ed8c6e83a5
I20260620 12:28:47.011576 17403 raft_consensus.cc:2726] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:28:47.132553 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:47.132150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:60663)
0620 12:28:47.132398 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:47.132458 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
I20260620 12:28:47.314239 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:47.313855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:52111)
0620 12:28:47.314096 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:47.314149 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260620 12:28:47.979707 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:47.979172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:37439)
0620 12:28:47.979533 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:47.979610 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260620 12:28:48.133611 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:48.133214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:48687)
0620 12:28:48.133466 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:48.133517 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260620 12:28:48.315496 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:48.315076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:37967)
0620 12:28:48.315348 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:48.315404 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260620 12:28:48.980916 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:48.980471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:40283)
0620 12:28:48.980758 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:48.980820 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260620 12:28:49.134838 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:49.134305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:53199)
0620 12:28:49.134571 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:49.134635 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260620 12:28:49.316737 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:49.316282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:43087)
0620 12:28:49.316563 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:49.316634 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260620 12:28:49.982149 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:49.981693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:55995)
0620 12:28:49.981987 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:49.982049 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
I20260620 12:28:50.136076 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:50.135659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:39307)
0620 12:28:50.135911 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:50.135979 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260620 12:28:50.318089 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:50.317671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:46399)
0620 12:28:50.317937 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:50.317996 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260620 12:28:50.928074 17415 raft_consensus.cc:482] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:50.928200 17415 raft_consensus.cc:504] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d156f39eefa46dbac4936e1fbf64662" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42733 } } peers { permanent_uuid: "efc0c5775e8e41f2995f16ed8c6e83a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "6a7a764ab60844f9b4b011634b529269" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46325 } }
I20260620 12:28:50.928574 17415 leader_election.cc:288] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733), efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829)
W20260620 12:28:50.929347 16987 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d156f39eefa46dbac4936e1fbf64662 (127.0.98.2:42733): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
W20260620 12:28:50.929455 16985 leader_election.cc:341] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efc0c5775e8e41f2995f16ed8c6e83a5 (127.0.98.3:40829): Not found: Tablet not found: 62ef4b051c3446c5bf339b87ce62eda1
I20260620 12:28:50.929560 16985 leader_election.cc:302] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a7a764ab60844f9b4b011634b529269; no voters: 3d156f39eefa46dbac4936e1fbf64662, efc0c5775e8e41f2995f16ed8c6e83a5
I20260620 12:28:50.929714 17415 raft_consensus.cc:2726] T 62ef4b051c3446c5bf339b87ce62eda1 P 6a7a764ab60844f9b4b011634b529269 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:28:50.983367 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:50.982960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:54181)
0620 12:28:50.983218 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:50.983273 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260620 12:28:51.137290 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:51.136853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:33207)
0620 12:28:51.137132 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:51.137189 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260620 12:28:51.319636 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:51.319107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:46151)
0620 12:28:51.319491 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:51.319543 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":113,"threads_started":1}
I20260620 12:28:51.984627 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:51.984226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.1:47737)
0620 12:28:51.984472 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:51.984525 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260620 12:28:52.138541 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:52.138162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.2:53281)
0620 12:28:52.138393 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:52.138448 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260620 12:28:52.311935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 16929
I20260620 12:28:52.317812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 17102
I20260620 12:28:52.320990 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:52.320474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.98.3:59105)
0620 12:28:52.320774 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:52.320850 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":64,"threads_started":1}
I20260620 12:28:52.323462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 17236
2026-06-20T12:28:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59357 ms)
[----------] 30 tests from SecurityITest (241401 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:28:52 dist-test-slave-bwpp krb5kdc[17424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:28:52 dist-test-slave-bwpp krb5kdc[17424](info): set up 2 sockets
Jun 20 12:28:52 dist-test-slave-bwpp krb5kdc[17424](info): commencing operation
krb5kdc: starting...
W20260620 12:28:53.386457 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:28:53 dist-test-slave-bwpp krb5kdc[17424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:28:53.505631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:53.611831 17437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:53.612069 17437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:53.612107 17437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:53.615978 17437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:28:53.616021 17437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:53.616040 17437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:53.616056 17437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:28:53.616071 17437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:28:53.621053 17437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:53.622042 17437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:53.622961 17437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:53.631088 17443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:53.631079 17445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:53.631181 17437 server_base.cc:1034] running on GCE node
W20260620 12:28:53.631078 17442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:53.631875 17437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:53.632943 17437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:53.634101 17437 hybrid_clock.cc:648] HybridClock initialized: now 1781958533634085 us; error 30 us; skew 500 ppm
Jun 20 12:28:53 dist-test-slave-bwpp krb5kdc[17424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:53.636942 17437 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:28:53.637856 17437 webserver.cc:460] Webserver started at http://127.0.98.62:45133/ using document root <none> and password file <none>
I20260620 12:28:53.638388 17437 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:53.638460 17437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:53.638742 17437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:53.640698 17437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "0967a66742bd4cf9a02c2368e96a5e42"
format_stamp: "Formatted at 2026-06-20 12:28:53 on dist-test-slave-bwpp"
server_key: "b3a344bc5e1b8afd68ea3683a398853b"
server_key_iv: "c722c88ec7b9070565d54b65e959bdcc"
server_key_version: "encryptionkey@0"
I20260620 12:28:53.641269 17437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "0967a66742bd4cf9a02c2368e96a5e42"
format_stamp: "Formatted at 2026-06-20 12:28:53 on dist-test-slave-bwpp"
server_key: "b3a344bc5e1b8afd68ea3683a398853b"
server_key_iv: "c722c88ec7b9070565d54b65e959bdcc"
server_key_version: "encryptionkey@0"
I20260620 12:28:53.644881 17437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:28:53.647258 17452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:53.648459 17437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:28:53.648604 17437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "0967a66742bd4cf9a02c2368e96a5e42"
format_stamp: "Formatted at 2026-06-20 12:28:53 on dist-test-slave-bwpp"
server_key: "b3a344bc5e1b8afd68ea3683a398853b"
server_key_iv: "c722c88ec7b9070565d54b65e959bdcc"
server_key_version: "encryptionkey@0"
I20260620 12:28:53.648715 17437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:53.660141 17437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:53.663633 17437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:53.663842 17437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:53.670774 17437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43997
I20260620 12:28:53.670789 17504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43997 every 8 connection(s)
I20260620 12:28:53.671751 17437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:28:53.674409 17505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:53.679796 17505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42: Bootstrap starting.
I20260620 12:28:53.681353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 17437
I20260620 12:28:53.681460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:28:53.681757 392 external_mini_cluster.cc:1428] Setting key 99896e967431a0d742c01ca989b2af11
I20260620 12:28:53.682330 17505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:53.683231 17505 log.cc:824] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:53.685367 17505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42: No bootstrap required, opened a new log
I20260620 12:28:53.689689 17505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967a66742bd4cf9a02c2368e96a5e42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:28:53.690017 17505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:53.690078 17505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0967a66742bd4cf9a02c2368e96a5e42, State: Initialized, Role: FOLLOWER
I20260620 12:28:53.690603 17505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967a66742bd4cf9a02c2368e96a5e42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:28:53.690827 17505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:28:53.690896 17505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:28:53.690997 17505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 12:28:53 dist-test-slave-bwpp krb5kdc[17424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:53.691957 17505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967a66742bd4cf9a02c2368e96a5e42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:28:53.692296 17505 leader_election.cc:302] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0967a66742bd4cf9a02c2368e96a5e42; no voters:
I20260620 12:28:53.692631 17505 leader_election.cc:288] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:28:53.692754 17510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:53.693001 17510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [term 1 LEADER]: Becoming Leader. State: Replica: 0967a66742bd4cf9a02c2368e96a5e42, State: Running, Role: LEADER
I20260620 12:28:53.693387 17510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967a66742bd4cf9a02c2368e96a5e42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:28:53.693926 17505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:28:53.694751 17512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0967a66742bd4cf9a02c2368e96a5e42. Latest consensus state: current_term: 1 leader_uuid: "0967a66742bd4cf9a02c2368e96a5e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967a66742bd4cf9a02c2368e96a5e42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } } }
I20260620 12:28:53.694877 17512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:53.695165 17511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0967a66742bd4cf9a02c2368e96a5e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967a66742bd4cf9a02c2368e96a5e42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } } }
I20260620 12:28:53.695380 17518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:28:53.695405 17511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42 [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:53.697937 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:53.682891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.0.98.62:43997)
0620 12:28:53.683282 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:53.683318 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:53.683363 (+ 45us) server_negotiation.cc:407] Connection header received
0620 12:28:53.684206 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:53.684226 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:53.684539 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:53.684823 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:53.685847 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:53.686919 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:53.688150 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:53.688618 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:53.688740 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:53.691831 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:53.691840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:53.691859 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:53.691886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:53.694074 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:53.694596 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:53.694602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:53.694607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:53.694802 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:53.695164 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:53.695169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:53.695173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:53.695688 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:53.695890 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:53.696641 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 12:28:53.696966 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260620 12:28:53.698608 17518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:28:53.703715 17518 catalog_manager.cc:1347] Generated new cluster ID: 5533eccc7b284f4f87e658166ce4d8da
I20260620 12:28:53.703778 17518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:28:53.721326 17518 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:28:53.722316 17518 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:28:53.727041 17518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42: Generated new TSK 0
I20260620 12:28:53.727736 17518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:28:53.757773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:53.870035 17531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:53.870316 17531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:53.870395 17531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:53.874383 17531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:53.874471 17531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:53.874579 17531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:28:53.879715 17531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:53.880708 17531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:53.881762 17531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:53.891206 17537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:53.891223 17536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:53.891207 17539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:53.891510 17531 server_base.cc:1034] running on GCE node
I20260620 12:28:53.891971 17531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:53.892580 17531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:53.893766 17531 hybrid_clock.cc:648] HybridClock initialized: now 1781958533893738 us; error 46 us; skew 500 ppm
Jun 20 12:28:53 dist-test-slave-bwpp krb5kdc[17424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:53.896869 17531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:28:53.897714 17531 webserver.cc:460] Webserver started at http://127.0.98.1:38715/ using document root <none> and password file <none>
I20260620 12:28:53.898252 17531 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:53.898320 17531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:53.898557 17531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:53.900525 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe18d3e2abd443f3b3e89a33bfdd6953"
format_stamp: "Formatted at 2026-06-20 12:28:53 on dist-test-slave-bwpp"
server_key: "ad070f1be09d617da2ad988509d9628d"
server_key_iv: "5d638395b72afa05ffab7734d28a1066"
server_key_version: "encryptionkey@0"
I20260620 12:28:53.901108 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe18d3e2abd443f3b3e89a33bfdd6953"
format_stamp: "Formatted at 2026-06-20 12:28:53 on dist-test-slave-bwpp"
server_key: "ad070f1be09d617da2ad988509d9628d"
server_key_iv: "5d638395b72afa05ffab7734d28a1066"
server_key_version: "encryptionkey@0"
I20260620 12:28:53.904994 17531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:28:53.907461 17546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:53.908650 17531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:28:53.908789 17531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "fe18d3e2abd443f3b3e89a33bfdd6953"
format_stamp: "Formatted at 2026-06-20 12:28:53 on dist-test-slave-bwpp"
server_key: "ad070f1be09d617da2ad988509d9628d"
server_key_iv: "5d638395b72afa05ffab7734d28a1066"
server_key_version: "encryptionkey@0"
I20260620 12:28:53.908897 17531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:53.922259 17531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:53.925026 17531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:53.925212 17531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:53.925740 17531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:53.926525 17531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:53.926596 17531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:53.926685 17531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:53.926723 17531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:53.935523 17531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37015
I20260620 12:28:53.935703 17659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37015 every 8 connection(s)
I20260620 12:28:53.936481 17531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:28:53.944092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 17531
I20260620 12:28:53.944245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:28:53.944555 392 external_mini_cluster.cc:1428] Setting key 872d2531cab74b578887b2af23f348a7
Jun 20 12:28:53 dist-test-slave-bwpp krb5kdc[17424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:53.950568 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:53.938247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51921 (local address 127.0.98.62:43997)
0620 12:28:53.938380 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:53.938384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:53.939249 (+ 865us) server_negotiation.cc:407] Connection header received
0620 12:28:53.940553 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:53.940556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:53.940614 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:53.940693 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:53.941810 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:53.942352 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:53.943551 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:53.943818 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:53.943933 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:53.946680 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:53.946687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:53.946693 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:53.946727 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:53.948247 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:53.948870 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:53.948873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:53.948874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:53.948973 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:53.949342 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:53.949348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:53.949353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:53.949599 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:53.949706 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:53.950182 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 12:28:53.950325 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:28:53.951308 17662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:53.938613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43997 (local address 127.0.98.1:51921)
0620 12:28:53.939108 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:53.939147 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:53.940340 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:53.940859 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:53.940866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:53.941196 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:53.941652 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:53.941665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:53.942481 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:53.942485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:53.943424 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:53.943434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:53.943968 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:53.943972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:53.944135 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:53.944818 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:53.944834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:53.946498 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:53.948366 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:53.948376 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:53.948391 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:53.948771 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:53.949102 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:53.949105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:53.949106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:53.949242 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:53.949702 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:53.949706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:53.949920 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 12:28:53.950094 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 12:28:53.950407 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":113,"threads_started":1}
I20260620 12:28:53.952737 17660 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43997
I20260620 12:28:53.953004 17660 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:53.953580 17660 heartbeater.cc:511] Master 127.0.98.62:43997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:28:53.955222 17469 ts_manager.cc:194] Registered new tserver with Master: fe18d3e2abd443f3b3e89a33bfdd6953 (127.0.98.1:37015)
I20260620 12:28:53.956321 17469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:28:53.973598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:54.080510 17665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:54.080747 17665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:54.080824 17665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:54.084743 17665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:54.084791 17665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:54.084846 17665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:28:54.089982 17665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:54.090975 17665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:54.091925 17665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:54.101327 17670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:54.101327 17671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:54.101404 17673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:54.101483 17665 server_base.cc:1034] running on GCE node
I20260620 12:28:54.101977 17665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:54.102557 17665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:54.103742 17665 hybrid_clock.cc:648] HybridClock initialized: now 1781958534103727 us; error 27 us; skew 500 ppm
Jun 20 12:28:54 dist-test-slave-bwpp krb5kdc[17424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:54.106467 17665 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:28:54.107331 17665 webserver.cc:460] Webserver started at http://127.0.98.2:46247/ using document root <none> and password file <none>
I20260620 12:28:54.107820 17665 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:54.107878 17665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:54.108032 17665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:54.109889 17665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "0305d28e35d9483c9c2de8c6934cb65d"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "fa041d6e7dda0584f268e3040d737773"
server_key_iv: "3987cecfc86637108c7a951d385c2297"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.110389 17665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0305d28e35d9483c9c2de8c6934cb65d"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "fa041d6e7dda0584f268e3040d737773"
server_key_iv: "3987cecfc86637108c7a951d385c2297"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.113969 17665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 12:28:54.116237 17680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:54.117386 17665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:28:54.117525 17665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "0305d28e35d9483c9c2de8c6934cb65d"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "fa041d6e7dda0584f268e3040d737773"
server_key_iv: "3987cecfc86637108c7a951d385c2297"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.117632 17665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:54.129981 17665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:54.132979 17665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:54.133165 17665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:54.133680 17665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:54.134413 17665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:54.134491 17665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:54.134558 17665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:54.134608 17665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:54.144181 17665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37159
I20260620 12:28:54.144325 17793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37159 every 8 connection(s)
I20260620 12:28:54.145355 17665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:28:54.149776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 17665
I20260620 12:28:54.149916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:28:54.150241 392 external_mini_cluster.cc:1428] Setting key d02e374457f02faed842c92e27595d59
Jun 20 12:28:54 dist-test-slave-bwpp krb5kdc[17424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:28:54.161860 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.149772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59065 (local address 127.0.98.62:43997)
0620 12:28:54.149904 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.149912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.149931 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:28:54.151096 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.151099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.151142 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.151377 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:54.152954 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.153669 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.154886 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.155106 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.155217 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.158073 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:54.158077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:54.158080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:54.158107 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:54.159875 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:54.160476 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:54.160482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:54.160485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:54.160575 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:54.160886 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:54.160888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:54.160889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:54.161058 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:54.161161 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.161644 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.161744 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:28:54.162899 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.148721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43997 (local address 127.0.98.2:59065)
0620 12:28:54.149246 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.149303 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.150892 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.151549 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:54.151560 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:54.152058 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:54.152799 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.152821 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.153802 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.153805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.154759 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.154776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.155263 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.155267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.155440 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.156092 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:54.156110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:54.157895 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:54.160012 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:54.160017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:54.160027 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:54.160372 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:54.160662 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:54.160665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:54.160666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:54.160804 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:54.161174 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:54.161183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:54.161439 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 12:28:54.161593 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 12:28:54.161839 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260620 12:28:54.164077 17794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43997
I20260620 12:28:54.164309 17794 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:54.164796 17794 heartbeater.cc:511] Master 127.0.98.62:43997 requested a full tablet report, sending...
I20260620 12:28:54.165882 17469 ts_manager.cc:194] Registered new tserver with Master: 0305d28e35d9483c9c2de8c6934cb65d (127.0.98.2:37159)
I20260620 12:28:54.166617 17469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:28:54.177541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:54.284909 17799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:54.285193 17799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:54.285269 17799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:54.289182 17799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:54.289269 17799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:54.289358 17799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:28:54.294576 17799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:54.295713 17799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:54.296705 17799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:54.306368 17804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:54.306392 17807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:54.306368 17805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:54.306666 17799 server_base.cc:1034] running on GCE node
I20260620 12:28:54.307062 17799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:54.307682 17799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:54.308844 17799 hybrid_clock.cc:648] HybridClock initialized: now 1781958534308839 us; error 38 us; skew 500 ppm
Jun 20 12:28:54 dist-test-slave-bwpp krb5kdc[17424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:54.311635 17799 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:28:54.312608 17799 webserver.cc:460] Webserver started at http://127.0.98.3:41427/ using document root <none> and password file <none>
I20260620 12:28:54.313156 17799 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:54.313262 17799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:54.313514 17799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:54.315610 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb616086c2fe458ab136755faa006302"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "a26e383751c80353d851af038ac38fe0"
server_key_iv: "1900532053a5c0d71f74109ddf6cae17"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.316174 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb616086c2fe458ab136755faa006302"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "a26e383751c80353d851af038ac38fe0"
server_key_iv: "1900532053a5c0d71f74109ddf6cae17"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.319815 17799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:28:54.322146 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:54.323451 17799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:28:54.323577 17799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "bb616086c2fe458ab136755faa006302"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "a26e383751c80353d851af038ac38fe0"
server_key_iv: "1900532053a5c0d71f74109ddf6cae17"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.323679 17799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:54.343664 17799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:54.346506 17799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:54.346728 17799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:54.347296 17799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:54.348093 17799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:54.348140 17799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:54.348212 17799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:54.348249 17799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:54.357915 17799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45837
I20260620 12:28:54.357936 17927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45837 every 8 connection(s)
I20260620 12:28:54.358799 17799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:28:54.363250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 17799
I20260620 12:28:54.363386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:28:54.363718 392 external_mini_cluster.cc:1428] Setting key 8844121d7be22979f27b8529a0e9a5ca
Jun 20 12:28:54 dist-test-slave-bwpp krb5kdc[17424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:54.372005 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.360580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60945 (local address 127.0.98.62:43997)
0620 12:28:54.360744 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.360749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.361568 (+ 819us) server_negotiation.cc:407] Connection header received
0620 12:28:54.362429 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.362432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.362530 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.362627 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:54.364255 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.364893 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.366270 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.366488 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.366569 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.368931 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:54.368934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:54.368936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:54.368958 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:54.370209 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:54.370739 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:54.370742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:54.370743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:54.370816 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:54.371113 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:54.371115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:54.371116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:54.371290 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:54.371365 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.371802 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.371890 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 12:28:54.372762 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.360936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43997 (local address 127.0.98.3:60945)
0620 12:28:54.361411 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.361453 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.362236 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.362857 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:54.362863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:54.363403 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:54.364086 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.364103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.365034 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.365038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.366150 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.366160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.366617 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.366620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.366856 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.367381 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:54.367396 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:54.368786 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:54.370318 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:54.370323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:54.370333 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:54.370582 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:54.370898 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:54.370900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:54.370901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:54.371029 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:54.371375 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:54.371380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:54.371602 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 12:28:54.371750 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 12:28:54.371980 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":114,"threads_started":1}
I20260620 12:28:54.373840 17928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43997
I20260620 12:28:54.374034 17928 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:54.374426 17928 heartbeater.cc:511] Master 127.0.98.62:43997 requested a full tablet report, sending...
I20260620 12:28:54.375396 17469 ts_manager.cc:194] Registered new tserver with Master: bb616086c2fe458ab136755faa006302 (127.0.98.3:45837)
I20260620 12:28:54.375890 17469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60945
I20260620 12:28:54.377393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:28:54.386286 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.379415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.0.98.62:43997)
0620 12:28:54.379558 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.379563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.379790 (+ 227us) server_negotiation.cc:407] Connection header received
0620 12:28:54.379908 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.379910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.379945 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.380009 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:54.380713 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.381172 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.381988 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.382180 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.382249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.383526 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:54.383529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:54.383531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:54.383551 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:54.384902 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:54.385265 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:54.385267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:54.385268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:54.385339 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:54.385561 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:54.385563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:54.385564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:54.385744 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:54.385847 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.386019 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.386099 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:28:54.389022 17469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:28:54.391968 17469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:28:54.406351 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.400105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.0.98.2:37159)
0620 12:28:54.401095 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.401108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.401132 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:28:54.401207 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.401213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.401409 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.401717 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:54.402078 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.403324 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.404658 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.405500 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.405618 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.405761 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.406078 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.406181 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":129,"threads_started":1}
I20260620 12:28:54.406409 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.399983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37159 (local address 127.0.0.1:45102)
0620 12:28:54.400579 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.400626 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.400751 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.401590 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:54.401596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:54.401644 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:54.401952 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.401960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.403497 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.403502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.404552 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.404564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.405643 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.405647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.405855 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.405921 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 12:28:54.406033 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:28:54.406228 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":374,"spinlock_wait_cycles":7552,"thread_start_us":64,"threads_started":1}
I20260620 12:28:54.406976 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.401130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.0.98.1:37015)
0620 12:28:54.401415 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.401426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.401450 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:28:54.401530 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.401535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.401758 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.401933 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:54.402130 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.403302 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.404803 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.405531 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.405626 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.405764 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.406722 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.406821 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":119,"threads_started":1}
I20260620 12:28:54.407132 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.400332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45837 (local address 127.0.0.1:59750)
0620 12:28:54.400807 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.400821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.400914 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.401590 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:54.401596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:54.401643 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:54.401782 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.401790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.403267 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.403271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.404679 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.404690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.406743 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.406747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.406843 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.406866 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:28:54.406917 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 12:28:54.406985 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":6,"spinlock_wait_cycles":5248,"thread_start_us":81,"threads_started":1}
I20260620 12:28:54.406409 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.400070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37015 (local address 127.0.0.1:33748)
0620 12:28:54.400737 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.400750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.400825 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.401906 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:54.401909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:54.401923 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:54.402047 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.402056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.403436 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.403447 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.404679 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.404690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.405643 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.405647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.405831 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.405921 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 12:28:54.406033 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:28:54.406223 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":148,"thread_start_us":92,"threads_started":1}
I20260620 12:28:54.407819 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.400737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.0.98.3:45837)
0620 12:28:54.401095 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.401108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.401132 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:28:54.401207 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.401213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.401409 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.402121 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:54.402257 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.403137 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.405830 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.406577 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.407414 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.407543 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.407591 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.407669 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":145,"threads_started":1}
I20260620 12:28:54.409556 17728 tablet_service.cc:1467] Processing CreateTablet for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2 (DEFAULT_TABLE table=test-table [id=41ddc947058040839b4f4f1fd51aacb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:54.409922 17862 tablet_service.cc:1467] Processing CreateTablet for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2 (DEFAULT_TABLE table=test-table [id=41ddc947058040839b4f4f1fd51aacb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:54.410128 17594 tablet_service.cc:1467] Processing CreateTablet for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2 (DEFAULT_TABLE table=test-table [id=41ddc947058040839b4f4f1fd51aacb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:54.410748 17728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:54.410789 17862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:54.411068 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:54.416795 17945 tablet_bootstrap.cc:492] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Bootstrap starting.
I20260620 12:28:54.418303 17946 tablet_bootstrap.cc:492] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: Bootstrap starting.
I20260620 12:28:54.418954 17945 tablet_bootstrap.cc:654] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:54.418973 17947 tablet_bootstrap.cc:492] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Bootstrap starting.
I20260620 12:28:54.419991 17945 log.cc:824] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:54.420435 17946 tablet_bootstrap.cc:654] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:54.420994 17947 tablet_bootstrap.cc:654] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:54.421768 17946 log.cc:824] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:54.422164 17947 log.cc:824] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:54.422252 17945 tablet_bootstrap.cc:492] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: No bootstrap required, opened a new log
I20260620 12:28:54.422451 17945 ts_tablet_manager.cc:1397] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:28:54.423594 17946 tablet_bootstrap.cc:492] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: No bootstrap required, opened a new log
I20260620 12:28:54.423789 17946 ts_tablet_manager.cc:1397] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:28:54.424160 17947 tablet_bootstrap.cc:492] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: No bootstrap required, opened a new log
I20260620 12:28:54.424335 17947 ts_tablet_manager.cc:1397] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:28:54.426388 17945 raft_consensus.cc:348] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.426688 17945 raft_consensus.cc:374] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:54.426754 17945 raft_consensus.cc:729] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb616086c2fe458ab136755faa006302, State: Initialized, Role: FOLLOWER
I20260620 12:28:54.426780 17946 raft_consensus.cc:348] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.427035 17946 raft_consensus.cc:374] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:54.427102 17946 raft_consensus.cc:729] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe18d3e2abd443f3b3e89a33bfdd6953, State: Initialized, Role: FOLLOWER
I20260620 12:28:54.427273 17945 consensus_queue.cc:260] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.427527 17946 consensus_queue.cc:260] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.428143 17660 heartbeater.cc:503] Master 127.0.98.62:43997 was elected leader, sending a full tablet report...
I20260620 12:28:54.428296 17945 ts_tablet_manager.cc:1428] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:28:54.428629 17928 heartbeater.cc:503] Master 127.0.98.62:43997 was elected leader, sending a full tablet report...
I20260620 12:28:54.428925 17947 raft_consensus.cc:348] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.429159 17947 raft_consensus.cc:374] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:54.429215 17947 raft_consensus.cc:729] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0305d28e35d9483c9c2de8c6934cb65d, State: Initialized, Role: FOLLOWER
I20260620 12:28:54.429606 17947 consensus_queue.cc:260] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.429760 17946 ts_tablet_manager.cc:1428] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:28:54.430521 17947 ts_tablet_manager.cc:1428] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260620 12:28:54.430891 17794 heartbeater.cc:503] Master 127.0.98.62:43997 was elected leader, sending a full tablet report...
W20260620 12:28:54.437485 17661 tablet.cc:2369] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:28:54.581023 17953 raft_consensus.cc:482] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:54.581290 17953 raft_consensus.cc:504] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.582499 17953 leader_election.cc:288] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe18d3e2abd443f3b3e89a33bfdd6953 (127.0.98.1:37015), bb616086c2fe458ab136755faa006302 (127.0.98.3:45837)
I20260620 12:28:54.587037 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.582803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37015 (local address 127.0.98.2:38717)
0620 12:28:54.582950 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.582986 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.583116 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.583483 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:54.583489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:54.583519 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:54.583652 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.583659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.584772 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.584775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.585908 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.585917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.586568 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.586572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.586697 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.586731 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:28:54.586792 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 12:28:54.586875 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":33}
I20260620 12:28:54.587052 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.583108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38717 (local address 127.0.98.1:37015)
0620 12:28:54.583230 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.583234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.583247 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:28:54.583295 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.583297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.583334 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.583399 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:54.583818 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.584605 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.586041 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.586439 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.586556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.586619 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.586839 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.586919 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:28:54.587067 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.582878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45837 (local address 127.0.98.2:48591)
0620 12:28:54.583317 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.583329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.583462 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.583736 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:54.583742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:54.583764 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:54.583880 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.583885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.584856 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.584862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.585931 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:54.585941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.586596 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.586599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:54.586746 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.586763 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:28:54.586861 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 12:28:54.586923 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":20,"thread_start_us":137,"threads_started":1}
I20260620 12:28:54.587077 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.582929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48591 (local address 127.0.98.3:45837)
0620 12:28:54.583091 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.583096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.583421 (+ 325us) server_negotiation.cc:407] Connection header received
0620 12:28:54.583581 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.583583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.583624 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.583682 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:54.583990 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.584712 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.586052 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.586454 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.586576 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.586636 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.586893 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.586976 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 12:28:54.587657 17614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b7c8d108fb4333b8d2d8bfbf7157e2" candidate_uuid: "0305d28e35d9483c9c2de8c6934cb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" is_pre_election: true
I20260620 12:28:54.587667 17882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b7c8d108fb4333b8d2d8bfbf7157e2" candidate_uuid: "0305d28e35d9483c9c2de8c6934cb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb616086c2fe458ab136755faa006302" is_pre_election: true
I20260620 12:28:54.587967 17882 raft_consensus.cc:2454] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0305d28e35d9483c9c2de8c6934cb65d in term 0.
I20260620 12:28:54.587968 17614 raft_consensus.cc:2454] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0305d28e35d9483c9c2de8c6934cb65d in term 0.
I20260620 12:28:54.588550 17683 leader_election.cc:302] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0305d28e35d9483c9c2de8c6934cb65d, bb616086c2fe458ab136755faa006302, fe18d3e2abd443f3b3e89a33bfdd6953; no voters:
I20260620 12:28:54.588865 17953 raft_consensus.cc:2781] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:28:54.588961 17953 raft_consensus.cc:482] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:28:54.589021 17953 raft_consensus.cc:3037] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:54.590080 17953 raft_consensus.cc:504] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.590421 17953 leader_election.cc:288] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [CANDIDATE]: Term 1 election: Requested vote from peers fe18d3e2abd443f3b3e89a33bfdd6953 (127.0.98.1:37015), bb616086c2fe458ab136755faa006302 (127.0.98.3:45837)
I20260620 12:28:54.590806 17614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b7c8d108fb4333b8d2d8bfbf7157e2" candidate_uuid: "0305d28e35d9483c9c2de8c6934cb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953"
I20260620 12:28:54.590828 17882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b7c8d108fb4333b8d2d8bfbf7157e2" candidate_uuid: "0305d28e35d9483c9c2de8c6934cb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb616086c2fe458ab136755faa006302"
I20260620 12:28:54.590945 17614 raft_consensus.cc:3037] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:54.590965 17882 raft_consensus.cc:3037] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:54.591969 17882 raft_consensus.cc:2454] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0305d28e35d9483c9c2de8c6934cb65d in term 1.
I20260620 12:28:54.591967 17614 raft_consensus.cc:2454] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0305d28e35d9483c9c2de8c6934cb65d in term 1.
I20260620 12:28:54.592298 17681 leader_election.cc:302] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0305d28e35d9483c9c2de8c6934cb65d, bb616086c2fe458ab136755faa006302; no voters:
I20260620 12:28:54.592490 17953 raft_consensus.cc:2781] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:54.592715 17953 raft_consensus.cc:686] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 1 LEADER]: Becoming Leader. State: Replica: 0305d28e35d9483c9c2de8c6934cb65d, State: Running, Role: LEADER
I20260620 12:28:54.593055 17953 consensus_queue.cc:237] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } }
I20260620 12:28:54.596307 17467 catalog_manager.cc:5495] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0305d28e35d9483c9c2de8c6934cb65d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0305d28e35d9483c9c2de8c6934cb65d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0305d28e35d9483c9c2de8c6934cb65d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 } health_report { overall_health: UNKNOWN } } }
W20260620 12:28:54.609727 17929 tablet.cc:2369] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:28:54.639083 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.635627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.0.98.2:37159)
0620 12:28:54.635751 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.635756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.635771 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:28:54.635950 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.635953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.636005 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.636113 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:28:54.636346 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.636840 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.637722 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.637942 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.638019 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.638264 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:28:54.638713 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:28:54.638805 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.638869 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.638916 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 12:28:54.644948 17882 raft_consensus.cc:1261] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 1 FOLLOWER]: Refusing update from remote peer 0305d28e35d9483c9c2de8c6934cb65d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:54.644995 17614 raft_consensus.cc:1261] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 1 FOLLOWER]: Refusing update from remote peer 0305d28e35d9483c9c2de8c6934cb65d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:54.645637 17955 consensus_queue.cc:1035] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb616086c2fe458ab136755faa006302" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:54.645795 17953 consensus_queue.cc:1035] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe18d3e2abd443f3b3e89a33bfdd6953" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:28:54.646577 17795 tablet.cc:2369] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:28:54.651711 17958 mvcc.cc:205] Tried to move back new op lower bound from 7298902157898805248 to 7298902157696012288. Current Snapshot: MvccSnapshot[applied={T|T < 7298902157898805248}]
I20260620 12:28:54.654003 17959 mvcc.cc:205] Tried to move back new op lower bound from 7298902157898805248 to 7298902157696012288. Current Snapshot: MvccSnapshot[applied={T|T < 7298902157898805248}]
I20260620 12:28:54.661108 17469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:54.661264 17469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:54.663601 17469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0967a66742bd4cf9a02c2368e96a5e42: Sending DeleteTablet for 3 replicas of tablet 83b7c8d108fb4333b8d2d8bfbf7157e2
I20260620 12:28:54.664377 17728 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45102
I20260620 12:28:54.664395 17594 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33748
I20260620 12:28:54.664387 17862 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59750
I20260620 12:28:54.664884 17967 tablet_replica.cc:330] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953: stopping tablet replica
I20260620 12:28:54.665186 17969 tablet_replica.cc:330] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: stopping tablet replica
I20260620 12:28:54.665222 17967 raft_consensus.cc:2229] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P fe18d3e2abd443f3b3e89a33bfdd6953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:54.665553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 17531
I20260620 12:28:54.667240 17969 raft_consensus.cc:2229] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:54.667609 17969 raft_consensus.cc:2258] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:54.667819 17968 tablet_replica.cc:330] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: stopping tablet replica
I20260620 12:28:54.668103 17968 raft_consensus.cc:2229] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:28:54.668504 17968 raft_consensus.cc:2258] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:54.669159 17969 ts_tablet_manager.cc:1905] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:54.672047 17969 ts_tablet_manager.cc:1918] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:54.672142 17969 log.cc:1196] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/83b7c8d108fb4333b8d2d8bfbf7157e2
I20260620 12:28:54.672484 17969 ts_tablet_manager.cc:1939] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P bb616086c2fe458ab136755faa006302: Deleting consensus metadata
W20260620 12:28:54.673772 17455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37015 (error 108)
I20260620 12:28:54.673839 17453 catalog_manager.cc:4841] TS bb616086c2fe458ab136755faa006302 (127.0.98.3:45837): tablet 83b7c8d108fb4333b8d2d8bfbf7157e2 (table test-table [id=41ddc947058040839b4f4f1fd51aacb3]) successfully deleted
I20260620 12:28:54.675506 17968 ts_tablet_manager.cc:1905] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:54.678263 17968 ts_tablet_manager.cc:1918] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:54.678373 17968 log.cc:1196] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/83b7c8d108fb4333b8d2d8bfbf7157e2
I20260620 12:28:54.678732 17968 ts_tablet_manager.cc:1939] T 83b7c8d108fb4333b8d2d8bfbf7157e2 P 0305d28e35d9483c9c2de8c6934cb65d: Deleting consensus metadata
W20260620 12:28:54.678818 17942 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:28:54.674509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37015 (local address 127.0.0.1:33758)
0620 12:28:54.674640 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.674678 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:54.674766 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:54.678723 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 12:28:54.678956 17455 catalog_manager.cc:4584] TS fe18d3e2abd443f3b3e89a33bfdd6953 (127.0.98.1:37015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83b7c8d108fb4333b8d2d8bfbf7157e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:28:54.679466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 17665
I20260620 12:28:54.685125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 17799
I20260620 12:28:54.685308 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.685015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37159 (local address 127.0.0.1:45124)
0620 12:28:54.685126 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:54.685225 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 12:28:54.691814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 17437
2026-06-20T12:28:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2367 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T12:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:28:54Z Disabled control of system clock
I20260620 12:28:54.720960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:28:54.841522 17974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:54.841797 17974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:54.841836 17974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:54.845943 17974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:28:54.846031 17974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:54.846053 17974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:54.846073 17974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:28:54.846092 17974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:28:54.850837 17974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:54.851845 17974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:54.853147 17974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:54.861442 17979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:54.861434 17980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:54.861644 17974 server_base.cc:1034] running on GCE node
W20260620 12:28:54.861429 17982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:54.862303 17974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:54.863543 17974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:54.864744 17974 hybrid_clock.cc:648] HybridClock initialized: now 1781958534864733 us; error 25 us; skew 500 ppm
I20260620 12:28:54.866515 17974 webserver.cc:460] Webserver started at http://127.0.98.62:37041/ using document root <none> and password file <none>
I20260620 12:28:54.867147 17974 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:54.867244 17974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:54.867450 17974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:54.869460 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "8337d29dc5c67766f94e12cda363b041"
server_key_iv: "e16d8e38f0f370e4438ff7bd7eb563ae"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.870044 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "8337d29dc5c67766f94e12cda363b041"
server_key_iv: "e16d8e38f0f370e4438ff7bd7eb563ae"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.873560 17974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:28:54.876047 17988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:54.877238 17974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:28:54.877380 17974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30"
format_stamp: "Formatted at 2026-06-20 12:28:54 on dist-test-slave-bwpp"
server_key: "8337d29dc5c67766f94e12cda363b041"
server_key_iv: "e16d8e38f0f370e4438ff7bd7eb563ae"
server_key_version: "encryptionkey@0"
I20260620 12:28:54.877490 17974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:54.886644 17974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:54.887271 17974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:54.887466 17974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:54.894048 17974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37055
I20260620 12:28:54.894068 18040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37055 every 8 connection(s)
I20260620 12:28:54.894965 17974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:28:54.896981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 17974
I20260620 12:28:54.897107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:28:54.897431 392 external_mini_cluster.cc:1428] Setting key a91df8b7efec5d4cd36438e789499a6b
I20260620 12:28:54.898578 18041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:54.903976 18041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30: Bootstrap starting.
I20260620 12:28:54.906114 18041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:54.906481 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:54.898902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.0.98.62:37055)
0620 12:28:54.899534 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:54.899549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:54.899583 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:28:54.900208 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:54.900224 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:54.900605 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:54.900945 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:54.901355 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.902277 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.903251 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:54.903720 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:54.903891 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:54.904782 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:54.904786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:54.904798 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:54.905165 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:54.905217 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:54.905237 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:54.905354 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:54.905500 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 12:28:54.905746 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":134,"threads_started":1}
I20260620 12:28:54.906948 18041 log.cc:824] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:54.908792 18041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30: No bootstrap required, opened a new log
I20260620 12:28:54.912113 18041 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 12:28:54.912343 18041 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:54.912417 18041 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe50f0dfd9fa4e5a9aff36cd94e74c30, State: Initialized, Role: FOLLOWER
I20260620 12:28:54.912844 18041 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 12:28:54.912986 18041 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:28:54.913067 18041 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:28:54.913159 18041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:54.914103 18041 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 12:28:54.914479 18041 leader_election.cc:302] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe50f0dfd9fa4e5a9aff36cd94e74c30; no voters:
I20260620 12:28:54.914812 18041 leader_election.cc:288] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:28:54.914922 18046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:54.915156 18046 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [term 1 LEADER]: Becoming Leader. State: Replica: fe50f0dfd9fa4e5a9aff36cd94e74c30, State: Running, Role: LEADER
I20260620 12:28:54.915436 18046 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260620 12:28:54.915951 18041 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:28:54.917057 18048 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe50f0dfd9fa4e5a9aff36cd94e74c30. Latest consensus state: current_term: 1 leader_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260620 12:28:54.917080 18047 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe50f0dfd9fa4e5a9aff36cd94e74c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260620 12:28:54.917179 18048 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:54.917186 18047 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30 [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:54.917569 18055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:28:54.920181 18055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:28:54.925017 18055 catalog_manager.cc:1347] Generated new cluster ID: 652a8f4b9fb14903bb8eb7d8bdf9df4e
I20260620 12:28:54.925078 18055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:28:54.940537 18055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:28:54.941434 18055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:28:54.948753 18055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30: Generated new TSK 0
I20260620 12:28:54.949414 18055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:28:54.954877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:28:55.065487 18065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:55.065728 18065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:55.065766 18065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:55.069640 18065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:55.069689 18065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:55.069741 18065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:28:55.074522 18065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:55.075532 18065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:55.076809 18065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:28:55.086398 18065 server_base.cc:1034] running on GCE node
W20260620 12:28:55.086333 18071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:55.086334 18073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:55.086350 18070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:55.087018 18065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:55.087636 18065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:55.088819 18065 hybrid_clock.cc:648] HybridClock initialized: now 1781958535088800 us; error 30 us; skew 500 ppm
I20260620 12:28:55.090919 18065 webserver.cc:460] Webserver started at http://127.0.98.1:35011/ using document root <none> and password file <none>
I20260620 12:28:55.091483 18065 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:55.091544 18065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:55.091750 18065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:55.093715 18065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd4bf16a93b14b5cb856dd11d8f5a488"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "b0625dd9a2d4b5c820678234b0f5e3de"
server_key_iv: "7dd8bd569352f12b01186d0819304ca4"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.094270 18065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd4bf16a93b14b5cb856dd11d8f5a488"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "b0625dd9a2d4b5c820678234b0f5e3de"
server_key_iv: "7dd8bd569352f12b01186d0819304ca4"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.097987 18065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:28:55.100519 18079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.101629 18065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:55.101761 18065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "cd4bf16a93b14b5cb856dd11d8f5a488"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "b0625dd9a2d4b5c820678234b0f5e3de"
server_key_iv: "7dd8bd569352f12b01186d0819304ca4"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.101869 18065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:55.119285 18065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:55.119907 18065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:55.120105 18065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:55.120698 18065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:55.121608 18065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:55.121678 18065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.121745 18065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:55.121794 18065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.130761 18065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44069
I20260620 12:28:55.130793 18192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44069 every 8 connection(s)
I20260620 12:28:55.131614 18065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:28:55.140654 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.133413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35729 (local address 127.0.98.62:37055)
0620 12:28:55.133543 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.133547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.134550 (+ 1003us) server_negotiation.cc:407] Connection header received
0620 12:28:55.136012 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.136015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.136077 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.136151 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:55.136995 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.137566 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.138700 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.138962 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.139078 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.140151 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:55.140154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:55.140157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:55.140170 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:55.140189 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:55.140206 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:55.140284 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.140433 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.140512 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:28:55.141052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 18065
I20260620 12:28:55.141173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:28:55.141500 392 external_mini_cluster.cc:1428] Setting key 9a4877f388fe9fe20a4da81e9adfc9f4
I20260620 12:28:55.141630 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.133797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.1:35729)
0620 12:28:55.134409 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.134450 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.135777 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.136311 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.136318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.136399 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:55.136847 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.136858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.137711 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.137715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.138562 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.138571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.139110 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.139114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.139278 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.139883 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:28:55.139884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:55.139931 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:28:55.139996 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:55.140285 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:55.140296 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:55.140305 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.140432 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.140679 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260620 12:28:55.142930 18193 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260620 12:28:55.143165 18193 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:55.143725 18193 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260620 12:28:55.143916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:28:55.145164 18004 ts_manager.cc:194] Registered new tserver with Master: cd4bf16a93b14b5cb856dd11d8f5a488 (127.0.98.1:44069)
I20260620 12:28:55.146275 18004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35729
W20260620 12:28:55.251240 18196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:55.251533 18196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:55.251596 18196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:55.255579 18196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:55.255657 18196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:55.255748 18196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:28:55.260579 18196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:55.261602 18196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:55.263001 18196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:55.272568 18202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:55.272614 18196 server_base.cc:1034] running on GCE node
W20260620 12:28:55.272562 18201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:55.272562 18204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:55.273170 18196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:55.273804 18196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:55.274984 18196 hybrid_clock.cc:648] HybridClock initialized: now 1781958535274969 us; error 28 us; skew 500 ppm
I20260620 12:28:55.276692 18196 webserver.cc:460] Webserver started at http://127.0.98.2:35845/ using document root <none> and password file <none>
I20260620 12:28:55.277259 18196 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:55.277351 18196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:55.277578 18196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:55.279621 18196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "d39db99d2ed1461984c7a912644a6b04"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "0e334ac115d41d4bee4fdab9b9da38c5"
server_key_iv: "ffbcdc69b9bab43eafb7c16fcb0db547"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.280180 18196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d39db99d2ed1461984c7a912644a6b04"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "0e334ac115d41d4bee4fdab9b9da38c5"
server_key_iv: "ffbcdc69b9bab43eafb7c16fcb0db547"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.283811 18196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:28:55.286054 18210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.287433 18196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:28:55.287572 18196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "d39db99d2ed1461984c7a912644a6b04"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "0e334ac115d41d4bee4fdab9b9da38c5"
server_key_iv: "ffbcdc69b9bab43eafb7c16fcb0db547"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.287680 18196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:55.316332 18196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:55.317178 18196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:55.317390 18196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:55.318063 18196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:55.319033 18196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:55.319108 18196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.319176 18196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:55.319226 18196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.327986 18196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38925
I20260620 12:28:55.328016 18323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38925 every 8 connection(s)
I20260620 12:28:55.328855 18196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:28:55.330896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 18196
I20260620 12:28:55.331032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:28:55.331344 392 external_mini_cluster.cc:1428] Setting key 241960eb3ffe3761c465f09393f012ef
I20260620 12:28:55.333591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:28:55.339829 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.330797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58853 (local address 127.0.98.62:37055)
0620 12:28:55.330969 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.330974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.332565 (+ 1591us) server_negotiation.cc:407] Connection header received
0620 12:28:55.333905 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.333908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.333951 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.334015 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:55.335267 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.335786 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.337632 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.337851 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.337935 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.339338 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:55.339341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:55.339343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:55.339355 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:55.339373 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:55.339382 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:55.339440 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.339654 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.339706 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:28:55.341089 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.331244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.2:58853)
0620 12:28:55.332003 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.332057 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.333638 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.334218 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.334230 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.334343 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:55.335103 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.335127 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.335923 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.335927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.337505 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.337520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.338020 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.338027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.338267 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.338996 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:28:55.339001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:55.339064 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:28:55.339157 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:55.339478 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:55.339490 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:55.339498 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.339636 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.339960 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":153,"threads_started":1}
I20260620 12:28:55.342216 18324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260620 12:28:55.342487 18324 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:55.343160 18324 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260620 12:28:55.344251 18004 ts_manager.cc:194] Registered new tserver with Master: d39db99d2ed1461984c7a912644a6b04 (127.0.98.2:38925)
I20260620 12:28:55.344844 18004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58853
W20260620 12:28:55.444314 18327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:55.444573 18327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:55.444638 18327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:55.448793 18327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:55.448887 18327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:55.448995 18327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:28:55.453809 18327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:55.454895 18327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:55.456218 18327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:55.465530 18335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:55.465549 18332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:55.465688 18327 server_base.cc:1034] running on GCE node
W20260620 12:28:55.465837 18333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:55.466194 18327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:55.466874 18327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:55.468035 18327 hybrid_clock.cc:648] HybridClock initialized: now 1781958535468020 us; error 26 us; skew 500 ppm
I20260620 12:28:55.469962 18327 webserver.cc:460] Webserver started at http://127.0.98.3:38461/ using document root <none> and password file <none>
I20260620 12:28:55.470520 18327 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:55.470582 18327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:55.470860 18327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:55.472790 18327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "de3fb3376bd141f1b06b5857fb784556"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "4cbf3969b9427e2a16fe7a6932ab5449"
server_key_iv: "0ffa01033b17cacd6b3c3801e7eb66e9"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.473330 18327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de3fb3376bd141f1b06b5857fb784556"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "4cbf3969b9427e2a16fe7a6932ab5449"
server_key_iv: "0ffa01033b17cacd6b3c3801e7eb66e9"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.476830 18327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:28:55.479136 18341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.480358 18327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:28:55.480475 18327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "de3fb3376bd141f1b06b5857fb784556"
format_stamp: "Formatted at 2026-06-20 12:28:55 on dist-test-slave-bwpp"
server_key: "4cbf3969b9427e2a16fe7a6932ab5449"
server_key_iv: "0ffa01033b17cacd6b3c3801e7eb66e9"
server_key_version: "encryptionkey@0"
I20260620 12:28:55.480607 18327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:55.495442 18327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:55.496058 18327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:55.496238 18327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:55.496783 18327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:55.497663 18327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:55.497707 18327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.497781 18327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:55.497816 18327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:55.506219 18327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40111
I20260620 12:28:55.506280 18454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40111 every 8 connection(s)
I20260620 12:28:55.507105 18327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:28:55.509856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 18327
I20260620 12:28:55.509968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:28:55.510233 392 external_mini_cluster.cc:1428] Setting key 66951343936854003cd4504318817e63
I20260620 12:28:55.516005 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.508808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57293 (local address 127.0.98.62:37055)
0620 12:28:55.508971 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.508976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.510002 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 12:28:55.511719 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.511723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.511774 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.511857 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:55.512745 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.513269 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.514331 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.514537 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.514611 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.515626 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:55.515628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:55.515631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:55.515642 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:55.515659 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:55.515667 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:55.515724 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.515854 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.515901 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:28:55.516819 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.509203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.3:57293)
0620 12:28:55.509837 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.509875 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.511463 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.512006 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.512012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.512091 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:55.512623 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.512636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.513372 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.513375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.514226 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.514235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.514703 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.514710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.514862 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.515393 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:28:55.515395 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:55.515432 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:28:55.515496 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:55.515739 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:55.515744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:55.515748 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.515835 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.516033 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260620 12:28:55.517891 18455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260620 12:28:55.518110 18455 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:55.518579 18455 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260620 12:28:55.519536 18004 ts_manager.cc:194] Registered new tserver with Master: de3fb3376bd141f1b06b5857fb784556 (127.0.98.3:40111)
I20260620 12:28:55.520077 18004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57293
I20260620 12:28:55.523974 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:28:55.529276 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.525779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.98.62:37055)
0620 12:28:55.525894 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.525897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.526081 (+ 184us) server_negotiation.cc:407] Connection header received
0620 12:28:55.526168 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.526170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.526202 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.526267 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:55.526544 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.527121 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.527900 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.528106 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.528193 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.528880 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:55.528883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:55.528885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:55.528897 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:55.528915 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:55.528924 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:55.528989 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.529083 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.529135 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:28:55.532030 18004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:28:55.534896 18004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:28:55.548947 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.543573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.0.98.3:40111)
0620 12:28:55.544028 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.544041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.544065 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:28:55.544135 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.544142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.544344 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.544508 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:55.544824 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.545994 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.547759 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.548272 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.548396 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.548579 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.548678 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.548775 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":127,"threads_started":1}
I20260620 12:28:55.548931 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.542944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40111 (local address 127.0.0.1:51134)
0620 12:28:55.543812 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.543826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.543904 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.544517 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.544523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.544545 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:55.544675 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.544682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.546084 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.546087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.547668 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.547676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.548395 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.548398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.548502 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.548536 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.548652 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.548772 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":226,"thread_start_us":76,"threads_started":1}
I20260620 12:28:55.549126 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.542833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44069 (local address 127.0.0.1:43766)
0620 12:28:55.543584 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.543629 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.543749 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.544768 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.544771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.544789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:55.544883 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.544891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.546255 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.546259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.547574 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.547584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.548586 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.548590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.548682 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.548704 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.548753 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.548830 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260620 12:28:55.549283 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.543231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38925 (local address 127.0.0.1:37096)
0620 12:28:55.544151 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.544164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.544251 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.544954 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.544960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.544981 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:55.545060 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.545064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.546015 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.546020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.547386 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.547404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.548670 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.548673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.548754 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.548773 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.548830 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.548884 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":210,"thread_start_us":97,"threads_started":1}
I20260620 12:28:55.549299 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.543649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.0.98.2:38925)
0620 12:28:55.544147 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.544160 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.544374 (+ 214us) server_negotiation.cc:407] Connection header received
0620 12:28:55.544469 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.544475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.544668 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.544807 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:55.545170 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.545885 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.547532 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.548312 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.548421 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.548620 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.548995 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.549110 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":134,"threads_started":1}
I20260620 12:28:55.550176 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.542950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.0.98.1:44069)
0620 12:28:55.544341 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.544351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.544374 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:28:55.544449 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.544456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.544629 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.544791 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:55.545018 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.546125 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.547707 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.548453 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.549673 (+ 1220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.549854 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.549904 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.550024 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1299,"thread_start_us":148,"threads_started":1}
I20260620 12:28:55.551923 18258 tablet_service.cc:1467] Processing CreateTablet for tablet 2a29972aab0b438596a7a3cba0ce59f6 (DEFAULT_TABLE table=test-table [id=4dde6db0b2444efdb5106464bae1d3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:55.552019 18389 tablet_service.cc:1467] Processing CreateTablet for tablet 2a29972aab0b438596a7a3cba0ce59f6 (DEFAULT_TABLE table=test-table [id=4dde6db0b2444efdb5106464bae1d3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:55.552886 18258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a29972aab0b438596a7a3cba0ce59f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:55.553000 18389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a29972aab0b438596a7a3cba0ce59f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:55.552664 18127 tablet_service.cc:1467] Processing CreateTablet for tablet 2a29972aab0b438596a7a3cba0ce59f6 (DEFAULT_TABLE table=test-table [id=4dde6db0b2444efdb5106464bae1d3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:55.553551 18127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a29972aab0b438596a7a3cba0ce59f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:55.558797 18472 tablet_bootstrap.cc:492] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: Bootstrap starting.
I20260620 12:28:55.560143 18473 tablet_bootstrap.cc:492] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: Bootstrap starting.
I20260620 12:28:55.562255 18473 tablet_bootstrap.cc:654] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:55.562254 18472 tablet_bootstrap.cc:654] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:55.562709 18474 tablet_bootstrap.cc:492] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Bootstrap starting.
I20260620 12:28:55.563165 18473 log.cc:824] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:55.563165 18472 log.cc:824] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:55.564673 18474 tablet_bootstrap.cc:654] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:55.565196 18473 tablet_bootstrap.cc:492] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: No bootstrap required, opened a new log
I20260620 12:28:55.565200 18472 tablet_bootstrap.cc:492] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: No bootstrap required, opened a new log
I20260620 12:28:55.565382 18473 ts_tablet_manager.cc:1397] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:28:55.565382 18472 ts_tablet_manager.cc:1397] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 12:28:55.565644 18474 log.cc:824] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:55.567255 18474 tablet_bootstrap.cc:492] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: No bootstrap required, opened a new log
I20260620 12:28:55.567454 18474 ts_tablet_manager.cc:1397] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:28:55.569734 18472 raft_consensus.cc:348] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.569742 18473 raft_consensus.cc:348] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.570024 18472 raft_consensus.cc:374] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:55.570024 18473 raft_consensus.cc:374] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:55.570083 18472 raft_consensus.cc:729] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39db99d2ed1461984c7a912644a6b04, State: Initialized, Role: FOLLOWER
I20260620 12:28:55.570083 18473 raft_consensus.cc:729] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4bf16a93b14b5cb856dd11d8f5a488, State: Initialized, Role: FOLLOWER
I20260620 12:28:55.570531 18472 consensus_queue.cc:260] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.570531 18473 consensus_queue.cc:260] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.570508 18474 raft_consensus.cc:348] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.570746 18474 raft_consensus.cc:374] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:55.570801 18474 raft_consensus.cc:729] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3fb3376bd141f1b06b5857fb784556, State: Initialized, Role: FOLLOWER
I20260620 12:28:55.571223 18324 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260620 12:28:55.571244 18474 consensus_queue.cc:260] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.571514 18473 ts_tablet_manager.cc:1428] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 12:28:55.571805 18193 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260620 12:28:55.572481 18474 ts_tablet_manager.cc:1428] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:28:55.572784 18455 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260620 12:28:55.572975 18472 ts_tablet_manager.cc:1428] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
W20260620 12:28:55.579727 18325 tablet.cc:2369] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:28:55.632743 18194 tablet.cc:2369] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:28:55.638911 18478 raft_consensus.cc:482] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:55.639128 18478 raft_consensus.cc:504] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.640184 18478 leader_election.cc:288] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd4bf16a93b14b5cb856dd11d8f5a488 (127.0.98.1:44069), de3fb3376bd141f1b06b5857fb784556 (127.0.98.3:40111)
I20260620 12:28:55.643950 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.640430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44069 (local address 127.0.98.2:35709)
0620 12:28:55.640617 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.640633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.640711 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.640996 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.641001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.641025 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:55.641159 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.641166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.641990 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.641993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.642822 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.642830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.643532 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.643535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.643641 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.643663 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.643735 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.643799 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 12:28:55.643990 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.640507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35709 (local address 127.0.98.1:44069)
0620 12:28:55.640643 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.640647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.640686 (+ 39us) server_negotiation.cc:407] Connection header received
0620 12:28:55.640816 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.640820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.640874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.640969 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:55.641283 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.641882 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.642962 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.643408 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.643490 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.643536 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.643765 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.643848 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:28:55.644081 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.640618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38711 (local address 127.0.98.3:40111)
0620 12:28:55.640730 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.640736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.640973 (+ 237us) server_negotiation.cc:407] Connection header received
0620 12:28:55.641109 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.641112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.641160 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.641242 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:55.641527 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.642021 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.643104 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.643523 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.643624 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.643679 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.643893 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.643958 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:28:55.644105 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.640430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40111 (local address 127.0.98.2:38711)
0620 12:28:55.640873 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.640887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:55.641003 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:55.641282 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:55.641285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:55.641301 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:55.641410 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.641416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.642132 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.642138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.642982 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:55.642991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.643657 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.643660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:55.643757 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.643782 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:28:55.643857 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:28:55.643919 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":37,"thread_start_us":127,"threads_started":1}
I20260620 12:28:55.644690 18409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a29972aab0b438596a7a3cba0ce59f6" candidate_uuid: "d39db99d2ed1461984c7a912644a6b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3fb3376bd141f1b06b5857fb784556" is_pre_election: true
I20260620 12:28:55.644755 18147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a29972aab0b438596a7a3cba0ce59f6" candidate_uuid: "d39db99d2ed1461984c7a912644a6b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" is_pre_election: true
I20260620 12:28:55.644994 18409 raft_consensus.cc:2454] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d39db99d2ed1461984c7a912644a6b04 in term 0.
I20260620 12:28:55.645035 18147 raft_consensus.cc:2454] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d39db99d2ed1461984c7a912644a6b04 in term 0.
I20260620 12:28:55.645510 18213 leader_election.cc:302] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cd4bf16a93b14b5cb856dd11d8f5a488, d39db99d2ed1461984c7a912644a6b04, de3fb3376bd141f1b06b5857fb784556; no voters:
I20260620 12:28:55.645761 18478 raft_consensus.cc:2781] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:28:55.645830 18478 raft_consensus.cc:482] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:28:55.645884 18478 raft_consensus.cc:3037] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:55.646924 18478 raft_consensus.cc:504] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.647282 18478 leader_election.cc:288] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [CANDIDATE]: Term 1 election: Requested vote from peers cd4bf16a93b14b5cb856dd11d8f5a488 (127.0.98.1:44069), de3fb3376bd141f1b06b5857fb784556 (127.0.98.3:40111)
I20260620 12:28:55.647609 18409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a29972aab0b438596a7a3cba0ce59f6" candidate_uuid: "d39db99d2ed1461984c7a912644a6b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3fb3376bd141f1b06b5857fb784556"
I20260620 12:28:55.647624 18147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a29972aab0b438596a7a3cba0ce59f6" candidate_uuid: "d39db99d2ed1461984c7a912644a6b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488"
I20260620 12:28:55.647728 18409 raft_consensus.cc:3037] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:55.647728 18147 raft_consensus.cc:3037] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:55.648941 18409 raft_consensus.cc:2454] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39db99d2ed1461984c7a912644a6b04 in term 1.
I20260620 12:28:55.648941 18147 raft_consensus.cc:2454] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39db99d2ed1461984c7a912644a6b04 in term 1.
I20260620 12:28:55.649266 18212 leader_election.cc:302] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cd4bf16a93b14b5cb856dd11d8f5a488, d39db99d2ed1461984c7a912644a6b04, de3fb3376bd141f1b06b5857fb784556; no voters:
I20260620 12:28:55.649468 18478 raft_consensus.cc:2781] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:55.649680 18478 raft_consensus.cc:686] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 1 LEADER]: Becoming Leader. State: Replica: d39db99d2ed1461984c7a912644a6b04, State: Running, Role: LEADER
I20260620 12:28:55.649999 18478 consensus_queue.cc:237] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } }
I20260620 12:28:55.652923 18005 catalog_manager.cc:5495] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 reported cstate change: term changed from 0 to 1, leader changed from <none> to d39db99d2ed1461984c7a912644a6b04 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d39db99d2ed1461984c7a912644a6b04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d39db99d2ed1461984c7a912644a6b04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38925 } health_report { overall_health: HEALTHY } } }
I20260620 12:28:55.668664 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.665599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.0.98.2:38925)
0620 12:28:55.665720 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:55.665723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:55.665734 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:28:55.665831 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:55.665833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:55.665880 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:55.665971 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:28:55.666169 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.666632 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.667499 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:55.667703 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:55.667775 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:55.668010 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:28:55.668376 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:28:55.668464 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:55.668513 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 12:28:55.668553 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:28:55.674053 18409 raft_consensus.cc:1261] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 1 FOLLOWER]: Refusing update from remote peer d39db99d2ed1461984c7a912644a6b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:55.674150 18147 raft_consensus.cc:1261] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 1 FOLLOWER]: Refusing update from remote peer d39db99d2ed1461984c7a912644a6b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:55.674813 18482 consensus_queue.cc:1035] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3fb3376bd141f1b06b5857fb784556" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:55.675033 18478 consensus_queue.cc:1035] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd4bf16a93b14b5cb856dd11d8f5a488" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:55.681149 18485 mvcc.cc:205] Tried to move back new op lower bound from 7298902162114478080 to 7298902162024824832. Current Snapshot: MvccSnapshot[applied={T|T < 7298902162114478080}]
I20260620 12:28:55.682548 18487 mvcc.cc:205] Tried to move back new op lower bound from 7298902162114478080 to 7298902162024824832. Current Snapshot: MvccSnapshot[applied={T|T < 7298902162114478080}]
I20260620 12:28:55.690452 18003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:55.690631 18003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:55.693086 18003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe50f0dfd9fa4e5a9aff36cd94e74c30: Sending DeleteTablet for 3 replicas of tablet 2a29972aab0b438596a7a3cba0ce59f6
I20260620 12:28:55.693796 18127 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a29972aab0b438596a7a3cba0ce59f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:55 UTC) from {username='slave'} at 127.0.0.1:43766
I20260620 12:28:55.693832 18389 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a29972aab0b438596a7a3cba0ce59f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:55 UTC) from {username='slave'} at 127.0.0.1:51134
I20260620 12:28:55.694036 18258 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a29972aab0b438596a7a3cba0ce59f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20260620 12:28:55.694308 18494 tablet_replica.cc:330] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488: stopping tablet replica
I20260620 12:28:55.694432 18496 tablet_replica.cc:330] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: stopping tablet replica
I20260620 12:28:55.694598 18494 raft_consensus.cc:2229] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:55.694736 18496 raft_consensus.cc:2229] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:28:55.694921 18494 raft_consensus.cc:2258] T 2a29972aab0b438596a7a3cba0ce59f6 P cd4bf16a93b14b5cb856dd11d8f5a488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:55.695099 18496 raft_consensus.cc:2258] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:55.695251 18495 tablet_replica.cc:330] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: stopping tablet replica
I20260620 12:28:55.695390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 18065
I20260620 12:28:55.695513 18495 raft_consensus.cc:2229] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:55.696892 18495 raft_consensus.cc:2258] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:55.698403 18495 ts_tablet_manager.cc:1905] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:55.701133 18495 ts_tablet_manager.cc:1918] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:55.701229 18495 log.cc:1196] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/2a29972aab0b438596a7a3cba0ce59f6
I20260620 12:28:55.701539 18495 ts_tablet_manager.cc:1939] T 2a29972aab0b438596a7a3cba0ce59f6 P de3fb3376bd141f1b06b5857fb784556: Deleting consensus metadata
W20260620 12:28:55.701892 17990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44069 (error 108)
I20260620 12:28:55.702672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 18196
I20260620 12:28:55.702703 18496 ts_tablet_manager.cc:1905] T 2a29972aab0b438596a7a3cba0ce59f6 P d39db99d2ed1461984c7a912644a6b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:55.703989 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.703631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44069 (local address 127.0.0.1:43770)
0620 12:28:55.703791 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.703874 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 12:28:55.704561 17991 catalog_manager.cc:4841] TS de3fb3376bd141f1b06b5857fb784556 (127.0.98.3:40111): tablet 2a29972aab0b438596a7a3cba0ce59f6 (table test-table [id=4dde6db0b2444efdb5106464bae1d3fe]) successfully deleted
W20260620 12:28:55.704818 17990 catalog_manager.cc:4584] TS cd4bf16a93b14b5cb856dd11d8f5a488 (127.0.98.1:44069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a29972aab0b438596a7a3cba0ce59f6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44069: connect: Connection refused (error 111)
I20260620 12:28:55.708858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 18327
I20260620 12:28:55.709966 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:55.709073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38925 (local address 127.0.0.1:37116)
0620 12:28:55.709790 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:55.709854 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619}
I20260620 12:28:55.714154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 17974
2026-06-20T12:28:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1022 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:28:55 dist-test-slave-bwpp krb5kdc[18500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:28:55 dist-test-slave-bwpp krb5kdc[18500](info): set up 2 sockets
Jun 20 12:28:55 dist-test-slave-bwpp krb5kdc[18500](info): commencing operation
krb5kdc: starting...
W20260620 12:28:56.782599 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:28:56 dist-test-slave-bwpp krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:28:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:28:56.901373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39387
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:57.008785 18513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:57.009020 18513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:57.009059 18513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:57.012895 18513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:28:57.012936 18513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:57.012956 18513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:57.012991 18513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:28:57.013006 18513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:28:57.017976 18513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:57.018893 18513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:57.019798 18513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:57.028100 18518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:57.028189 18513 server_base.cc:1034] running on GCE node
W20260620 12:28:57.028074 18521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:57.028083 18519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:57.028959 18513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:57.030017 18513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:57.031203 18513 hybrid_clock.cc:648] HybridClock initialized: now 1781958537031182 us; error 31 us; skew 500 ppm
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:57.033806 18513 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:28:57.034634 18513 webserver.cc:460] Webserver started at http://127.0.98.62:36003/ using document root <none> and password file <none>
I20260620 12:28:57.035239 18513 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:57.035331 18513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:57.035553 18513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:57.037534 18513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "1d11da2be444499c80d4eff2775af0e1"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "5eafb88cd54e03beb7a7b71dd6a064e3"
server_key_iv: "9dbea64bbb174ebd9b035b2e64331bdd"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.038100 18513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d11da2be444499c80d4eff2775af0e1"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "5eafb88cd54e03beb7a7b71dd6a064e3"
server_key_iv: "9dbea64bbb174ebd9b035b2e64331bdd"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.041661 18513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 12:28:57.044006 18528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.045197 18513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:28:57.045339 18513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "1d11da2be444499c80d4eff2775af0e1"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "5eafb88cd54e03beb7a7b71dd6a064e3"
server_key_iv: "9dbea64bbb174ebd9b035b2e64331bdd"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.045450 18513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:57.070253 18513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:57.073282 18513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:57.073490 18513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:57.080725 18513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39387
I20260620 12:28:57.080756 18580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39387 every 8 connection(s)
I20260620 12:28:57.081666 18513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:28:57.084329 18581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:57.087392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 18513
I20260620 12:28:57.087499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:28:57.087831 392 external_mini_cluster.cc:1428] Setting key 748592a6ff6429949d8d9d37fc8a4ec9
I20260620 12:28:57.090198 18581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1: Bootstrap starting.
I20260620 12:28:57.092834 18581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:57.093653 18581 log.cc:824] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:57.095750 18581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1: No bootstrap required, opened a new log
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:57.100173 18581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d11da2be444499c80d4eff2775af0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39387 } }
I20260620 12:28:57.100462 18581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:57.100522 18581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d11da2be444499c80d4eff2775af0e1, State: Initialized, Role: FOLLOWER
I20260620 12:28:57.101028 18581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d11da2be444499c80d4eff2775af0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39387 } }
I20260620 12:28:57.101204 18581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:28:57.101274 18581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:28:57.101372 18581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:57.102401 18581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d11da2be444499c80d4eff2775af0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39387 } }
I20260620 12:28:57.102758 18581 leader_election.cc:302] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d11da2be444499c80d4eff2775af0e1; no voters:
I20260620 12:28:57.103078 18581 leader_election.cc:288] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:28:57.103240 18586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:57.103338 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.089325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.0.98.62:39387)
0620 12:28:57.089740 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.089759 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.089800 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:28:57.090605 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.090627 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.091050 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.091423 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:57.092141 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.093631 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.094755 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.095417 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.095568 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.098147 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:57.098153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:57.098167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:57.098197 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:57.100220 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.100719 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.100724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.100729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.100862 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.101119 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.101122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.101123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.101640 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:57.101835 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.102045 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.102351 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260620 12:28:57.103452 18586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [term 1 LEADER]: Becoming Leader. State: Replica: 1d11da2be444499c80d4eff2775af0e1, State: Running, Role: LEADER
I20260620 12:28:57.103765 18586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d11da2be444499c80d4eff2775af0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39387 } }
I20260620 12:28:57.104251 18581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:28:57.104884 18586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d11da2be444499c80d4eff2775af0e1. Latest consensus state: current_term: 1 leader_uuid: "1d11da2be444499c80d4eff2775af0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d11da2be444499c80d4eff2775af0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39387 } } }
I20260620 12:28:57.105046 18586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:57.104884 18587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d11da2be444499c80d4eff2775af0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d11da2be444499c80d4eff2775af0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39387 } } }
I20260620 12:28:57.105142 18587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1 [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:57.105423 18594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:28:57.108187 18594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:28:57.113286 18594 catalog_manager.cc:1347] Generated new cluster ID: 8e178777ee1d4e33a2c5b318d8e0bf46
I20260620 12:28:57.113353 18594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:28:57.145848 18594 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:28:57.147104 18594 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:28:57.152642 18594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1: Generated new TSK 0
I20260620 12:28:57.153308 18594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:28:57.187399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39387
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:57.294629 18607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:57.294914 18607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:57.294987 18607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:57.298959 18607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:57.299031 18607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:57.299117 18607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:28:57.304584 18607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:57.305783 18607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:57.306840 18607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:57.316051 18613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:57.316040 18615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:57.316040 18612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:57.316169 18607 server_base.cc:1034] running on GCE node
I20260620 12:28:57.316689 18607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:57.317282 18607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:57.318467 18607 hybrid_clock.cc:648] HybridClock initialized: now 1781958537318452 us; error 28 us; skew 500 ppm
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:57.321579 18607 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:28:57.322381 18607 webserver.cc:460] Webserver started at http://127.0.98.1:35051/ using document root <none> and password file <none>
I20260620 12:28:57.322947 18607 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:57.323005 18607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:57.323160 18607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:57.324978 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "bed459d353ef4bf5895d23e7bab6e697"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "960f6468241f0eb8f60586bf45a5f709"
server_key_iv: "585290e3a91586d90a3889a3f3a837af"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.325513 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bed459d353ef4bf5895d23e7bab6e697"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "960f6468241f0eb8f60586bf45a5f709"
server_key_iv: "585290e3a91586d90a3889a3f3a837af"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.328972 18607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:28:57.331245 18622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.332389 18607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:28:57.332517 18607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "bed459d353ef4bf5895d23e7bab6e697"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "960f6468241f0eb8f60586bf45a5f709"
server_key_iv: "585290e3a91586d90a3889a3f3a837af"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.332619 18607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:57.350761 18607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:57.353741 18607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:57.353931 18607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:57.354516 18607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:57.355341 18607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:57.355392 18607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.355458 18607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:57.355489 18607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.364574 18607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36177
I20260620 12:28:57.364638 18735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36177 every 8 connection(s)
I20260620 12:28:57.365460 18607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:28:57.373503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 18607
I20260620 12:28:57.373633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:28:57.373947 392 external_mini_cluster.cc:1428] Setting key bc254e420e352492dc2fac956f8fdd23
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:28:57.386124 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.367405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58313 (local address 127.0.98.62:39387)
0620 12:28:57.367545 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.367548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.369723 (+ 2175us) server_negotiation.cc:407] Connection header received
0620 12:28:57.370706 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.370710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.370766 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.370873 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:57.372085 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.372623 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.374071 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.374447 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.374568 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.381990 (+ 7422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:57.381994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:57.381996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:57.382022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:57.383836 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.384473 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.384482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.384483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.384566 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.384872 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.384874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.384875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.385048 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:57.385124 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.385743 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.385922 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:28:57.387004 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.367815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39387 (local address 127.0.98.1:58313)
0620 12:28:57.369580 (+ 1765us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.369619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.370479 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.371042 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.371048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.371475 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:57.371940 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.371954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.372749 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.372755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.373931 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.373941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.374607 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.374611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.374828 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.379991 (+ 5163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:57.380018 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:57.381786 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:57.383985 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:57.383990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:57.384003 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:57.384262 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:57.384657 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:57.384659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:57.384660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:57.384790 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:57.385179 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:57.385185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:57.385523 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.385693 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.386015 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"thread_start_us":147,"threads_started":1}
I20260620 12:28:57.388218 18736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39387
I20260620 12:28:57.388506 18736 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:57.389094 18736 heartbeater.cc:511] Master 127.0.98.62:39387 requested a full tablet report, sending...
I20260620 12:28:57.390740 18545 ts_manager.cc:194] Registered new tserver with Master: bed459d353ef4bf5895d23e7bab6e697 (127.0.98.1:36177)
I20260620 12:28:57.391866 18545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:28:57.401399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39387
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:57.508900 18741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:57.509127 18741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:57.509207 18741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:57.513538 18741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:57.513590 18741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:57.513681 18741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:28:57.518721 18741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:57.519635 18741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:57.520560 18741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:57.529610 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:57.529621 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:57.529672 18741 server_base.cc:1034] running on GCE node
W20260620 12:28:57.529629 18749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:57.530403 18741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:57.531010 18741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:57.532187 18741 hybrid_clock.cc:648] HybridClock initialized: now 1781958537532179 us; error 30 us; skew 500 ppm
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:57.534785 18741 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:28:57.535560 18741 webserver.cc:460] Webserver started at http://127.0.98.2:35913/ using document root <none> and password file <none>
I20260620 12:28:57.536098 18741 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:57.536160 18741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:57.536317 18741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:57.538231 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "a86dd923c8084a8daf1702e641f03ff7"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "7c9993d7b0305859316a24f9a48bcb25"
server_key_iv: "65bcac11d52fffe9530f13d5ef393958"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.538776 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a86dd923c8084a8daf1702e641f03ff7"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "7c9993d7b0305859316a24f9a48bcb25"
server_key_iv: "65bcac11d52fffe9530f13d5ef393958"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.542138 18741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:28:57.544479 18756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.545603 18741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 12:28:57.545727 18741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "a86dd923c8084a8daf1702e641f03ff7"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "7c9993d7b0305859316a24f9a48bcb25"
server_key_iv: "65bcac11d52fffe9530f13d5ef393958"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.545828 18741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:57.566617 18741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:57.569557 18741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:57.569751 18741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:57.570335 18741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:57.571169 18741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:57.571216 18741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.571282 18741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:57.571314 18741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.580521 18741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44213
I20260620 12:28:57.580549 18869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44213 every 8 connection(s)
I20260620 12:28:57.581398 18741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:28:57.587416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 18741
I20260620 12:28:57.587565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:28:57.587845 392 external_mini_cluster.cc:1428] Setting key 56b3b9fd9a1a72731b400ed38ea1e10f
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:57.594906 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.583167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58319 (local address 127.0.98.62:39387)
0620 12:28:57.583302 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.583307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.584083 (+ 776us) server_negotiation.cc:407] Connection header received
0620 12:28:57.584995 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.584998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.585055 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.585162 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:57.586273 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.586861 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.588165 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.588500 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.588614 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.591337 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:57.591340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:57.591342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:57.591366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:57.592835 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.593397 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.593401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.593403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.593484 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.593819 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.593823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.593828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.594009 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:57.594114 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.594603 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.594748 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:28:57.595686 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.583481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39387 (local address 127.0.98.2:58319)
0620 12:28:57.583928 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.583968 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.584786 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.585337 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.585343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.585683 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:57.586125 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.586137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.586983 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.586987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.588042 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.588051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.588648 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.588652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.588821 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.589412 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:57.589428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:57.591159 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:57.592968 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:57.592973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:57.592986 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:57.593290 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:57.593583 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:57.593586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:57.593588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:57.593725 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:57.594124 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:57.594132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:57.594368 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.594548 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.594907 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:28:57.596822 18870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39387
I20260620 12:28:57.597077 18870 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:57.597596 18870 heartbeater.cc:511] Master 127.0.98.62:39387 requested a full tablet report, sending...
I20260620 12:28:57.598727 18545 ts_manager.cc:194] Registered new tserver with Master: a86dd923c8084a8daf1702e641f03ff7 (127.0.98.2:44213)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:28:57.599315 18545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:28:57.613670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39387
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:28:57.721649 18875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:57.721881 18875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:57.721922 18875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:57.725876 18875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:57.725927 18875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:57.726012 18875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:28:57.731180 18875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:57.732158 18875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:57.733160 18875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:57.742769 18883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:57.742834 18881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:57.742834 18880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:57.743232 18875 server_base.cc:1034] running on GCE node
I20260620 12:28:57.743618 18875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:57.744258 18875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:57.745448 18875 hybrid_clock.cc:648] HybridClock initialized: now 1781958537745433 us; error 26 us; skew 500 ppm
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:28:57.748138 18875 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:28:57.749053 18875 webserver.cc:460] Webserver started at http://127.0.98.3:34327/ using document root <none> and password file <none>
I20260620 12:28:57.749643 18875 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:57.749729 18875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:57.749964 18875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:57.752002 18875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "27cada6e8dfe4736b40eb250d0646f47"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "0b07fb3e6d49d9ca00d57c3cf2553119"
server_key_iv: "b94f15c14e953c625fd2e9e7e6237143"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.752545 18875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "27cada6e8dfe4736b40eb250d0646f47"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "0b07fb3e6d49d9ca00d57c3cf2553119"
server_key_iv: "b94f15c14e953c625fd2e9e7e6237143"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.756520 18875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 12:28:57.759393 18890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.760615 18875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 12:28:57.760766 18875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "27cada6e8dfe4736b40eb250d0646f47"
format_stamp: "Formatted at 2026-06-20 12:28:57 on dist-test-slave-bwpp"
server_key: "0b07fb3e6d49d9ca00d57c3cf2553119"
server_key_iv: "b94f15c14e953c625fd2e9e7e6237143"
server_key_version: "encryptionkey@0"
I20260620 12:28:57.760876 18875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:57.774302 18875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:57.777976 18875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:57.778213 18875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:57.778918 18875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:57.779832 18875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:57.779913 18875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.780009 18875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:57.780071 18875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:57.790000 18875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35933
I20260620 12:28:57.790026 19003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35933 every 8 connection(s)
I20260620 12:28:57.790973 18875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:28:57.799396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 18875
I20260620 12:28:57.799538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
Jun 20 12:28:57 dist-test-slave-bwpp krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:28:57.799872 392 external_mini_cluster.cc:1428] Setting key 212dd1144763f3e02aff5616d87f1b33
I20260620 12:28:57.803861 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.792778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33115 (local address 127.0.98.62:39387)
0620 12:28:57.792945 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.792950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.793704 (+ 754us) server_negotiation.cc:407] Connection header received
0620 12:28:57.794554 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.794556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.794602 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.794744 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:57.795881 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.796394 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.797486 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.797689 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.797762 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.800428 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:57.800432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:57.800435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:57.800473 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:57.802070 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.802520 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.802523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.802524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.802596 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.802959 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.802961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.802962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.803129 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:57.803206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.803654 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.803748 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:28:57.804819 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.793109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39387 (local address 127.0.98.3:33115)
0620 12:28:57.793552 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.793593 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.794365 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.794893 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.794899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.795231 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:57.795740 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.795765 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.796498 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.796500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.797378 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.797387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.797825 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.797827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.797963 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.798565 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:28:57.798581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:57.800239 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:57.802178 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:57.802183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:57.802193 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:57.802436 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:57.802721 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:28:57.802724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:28:57.802726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:28:57.802875 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:28:57.803229 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:57.803234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:57.803456 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.803608 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.803857 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260620 12:28:57.805941 19004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39387
I20260620 12:28:57.806182 19004 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:57.806692 19004 heartbeater.cc:511] Master 127.0.98.62:39387 requested a full tablet report, sending...
I20260620 12:28:57.807668 18545 ts_manager.cc:194] Registered new tserver with Master: 27cada6e8dfe4736b40eb250d0646f47 (127.0.98.3:35933)
I20260620 12:28:57.808214 18545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33115
I20260620 12:28:57.813587 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:28:57.822347 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.815472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.0.98.62:39387)
0620 12:28:57.815586 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.815589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.815743 (+ 154us) server_negotiation.cc:407] Connection header received
0620 12:28:57.815822 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.815824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.815855 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.815921 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:57.816526 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.817004 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.817771 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.817969 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.818041 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.819463 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:57.819466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:57.819468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:28:57.819489 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:57.820902 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.821351 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.821353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.821354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.821421 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:28:57.821679 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:28:57.821681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:28:57.821682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:28:57.821853 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:57.821959 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.822149 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.822215 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:28:57.824965 18545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:28:57.827963 18545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:28:57.842389 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.836098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36177 (local address 127.0.0.1:44142)
0620 12:28:57.836672 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.836722 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.836841 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.838168 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.838175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.838200 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:57.838344 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.838352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.839504 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.839509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.840780 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.840790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.841852 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.841858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.841973 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.842016 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.842127 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.842245 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":64,"threads_started":1}
I20260620 12:28:57.842453 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.836231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.98.1:36177)
0620 12:28:57.837166 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.837181 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.837203 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:28:57.837285 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.837292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.837533 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.837690 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:57.838481 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.839351 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.840900 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.841714 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.841830 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.841984 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.842171 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.842268 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":75,"threads_started":1}
I20260620 12:28:57.842559 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.836805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44213 (local address 127.0.0.1:37386)
0620 12:28:57.837930 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.837955 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.838048 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.838469 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.838472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.838488 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:57.838590 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.838595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.839915 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.839919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.841000 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.841007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.842152 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.842155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.842254 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.842278 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.842355 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.842424 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":86,"threads_started":1}
I20260620 12:28:57.842971 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.836985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.98.2:44213)
0620 12:28:57.837440 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.837454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.838048 (+ 594us) server_negotiation.cc:407] Connection header received
0620 12:28:57.838140 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.838144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.838332 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.838484 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:57.838705 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.839773 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.841109 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.841861 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.841953 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.842061 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.842725 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.842835 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":98,"threads_started":1}
I20260620 12:28:57.843873 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.836251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35933 (local address 127.0.0.1:41740)
0620 12:28:57.837305 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.837320 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.837406 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.838266 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.838269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.838288 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:57.838381 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.838392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.841095 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.841097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.841812 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.841823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.843436 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.843440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.843539 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.843562 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.843671 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.843754 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":94,"threads_started":1}
I20260620 12:28:57.844257 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.837187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.0.98.3:35933)
0620 12:28:57.837817 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.837831 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.837852 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:28:57.837927 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.837931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.838134 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.838275 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:57.838521 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.839751 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.841959 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.843299 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.843439 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.843578 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.844026 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.844120 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":115,"threads_started":1}
I20260620 12:28:57.845693 18669 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f6602b6194ebb9951bb76ce30db0b (DEFAULT_TABLE table=test-table [id=39af183d81ee4cbf9d6365abc6d69149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:57.845696 18804 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f6602b6194ebb9951bb76ce30db0b (DEFAULT_TABLE table=test-table [id=39af183d81ee4cbf9d6365abc6d69149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:57.846748 18669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f6602b6194ebb9951bb76ce30db0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:57.846755 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f6602b6194ebb9951bb76ce30db0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:57.847281 18938 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f6602b6194ebb9951bb76ce30db0b (DEFAULT_TABLE table=test-table [id=39af183d81ee4cbf9d6365abc6d69149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:57.848178 18938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f6602b6194ebb9951bb76ce30db0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:57.853103 19021 tablet_bootstrap.cc:492] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697: Bootstrap starting.
I20260620 12:28:57.854416 19022 tablet_bootstrap.cc:492] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Bootstrap starting.
I20260620 12:28:57.854621 19023 tablet_bootstrap.cc:492] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Bootstrap starting.
I20260620 12:28:57.856374 19023 tablet_bootstrap.cc:654] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:57.856580 19021 tablet_bootstrap.cc:654] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:57.856972 19022 tablet_bootstrap.cc:654] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:57.857467 19023 log.cc:824] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:57.857827 19022 log.cc:824] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:57.857827 19021 log.cc:824] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:57.859351 19023 tablet_bootstrap.cc:492] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: No bootstrap required, opened a new log
I20260620 12:28:57.859586 19023 ts_tablet_manager.cc:1397] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 12:28:57.859697 19022 tablet_bootstrap.cc:492] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: No bootstrap required, opened a new log
I20260620 12:28:57.859778 19021 tablet_bootstrap.cc:492] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697: No bootstrap required, opened a new log
I20260620 12:28:57.859871 19022 ts_tablet_manager.cc:1397] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:28:57.859965 19021 ts_tablet_manager.cc:1397] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 12:28:57.862625 19021 raft_consensus.cc:348] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.862953 19021 raft_consensus.cc:374] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:57.863010 19021 raft_consensus.cc:729] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed459d353ef4bf5895d23e7bab6e697, State: Initialized, Role: FOLLOWER
I20260620 12:28:57.863425 19021 consensus_queue.cc:260] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.863508 19023 raft_consensus.cc:348] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.863508 19022 raft_consensus.cc:348] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.863770 19022 raft_consensus.cc:374] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:57.863777 19023 raft_consensus.cc:374] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:57.863832 19022 raft_consensus.cc:729] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86dd923c8084a8daf1702e641f03ff7, State: Initialized, Role: FOLLOWER
I20260620 12:28:57.863832 19023 raft_consensus.cc:729] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cada6e8dfe4736b40eb250d0646f47, State: Initialized, Role: FOLLOWER
I20260620 12:28:57.864181 18736 heartbeater.cc:503] Master 127.0.98.62:39387 was elected leader, sending a full tablet report...
I20260620 12:28:57.864280 19023 consensus_queue.cc:260] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.864288 19022 consensus_queue.cc:260] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.864534 19021 ts_tablet_manager.cc:1428] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:28:57.865175 19022 ts_tablet_manager.cc:1428] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:28:57.865218 19004 heartbeater.cc:503] Master 127.0.98.62:39387 was elected leader, sending a full tablet report...
I20260620 12:28:57.865222 19023 ts_tablet_manager.cc:1428] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:28:57.866441 18870 heartbeater.cc:503] Master 127.0.98.62:39387 was elected leader, sending a full tablet report...
W20260620 12:28:57.866695 18737 tablet.cc:2369] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:28:57.873584 19027 raft_consensus.cc:482] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:57.873713 19027 raft_consensus.cc:504] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.874605 19027 leader_election.cc:288] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27cada6e8dfe4736b40eb250d0646f47 (127.0.98.3:35933), a86dd923c8084a8daf1702e641f03ff7 (127.0.98.2:44213)
I20260620 12:28:57.879518 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.874937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40179 (local address 127.0.98.3:35933)
0620 12:28:57.875092 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.875096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.875112 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:28:57.875263 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.875266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.875352 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.875432 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:57.875778 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.876580 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.878232 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.878874 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.878980 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.879041 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.879326 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.879394 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:28:57.879518 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.875282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56409 (local address 127.0.98.2:44213)
0620 12:28:57.875393 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.875397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.875593 (+ 196us) server_negotiation.cc:407] Connection header received
0620 12:28:57.875722 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.875725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.875767 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.875832 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:57.876063 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.876744 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.878232 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.878867 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.878981 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.879041 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.879340 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.879403 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 12:28:57.879539 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.874877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35933 (local address 127.0.98.1:40179)
0620 12:28:57.875029 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.875047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.875134 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.875550 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.875553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.875569 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:57.875684 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.875690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.876706 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.876709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.878106 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.878117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.879009 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.879013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.879184 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.879221 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.879300 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.879381 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":119}
I20260620 12:28:57.879546 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.875139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44213 (local address 127.0.98.1:56409)
0620 12:28:57.875516 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.875530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:57.875625 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:57.875865 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:57.875867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:57.875882 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:57.875965 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.875969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.876860 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.876864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.878114 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:57.878119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.879009 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.879013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:57.879157 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.879210 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 12:28:57.879295 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 12:28:57.879381 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":75,"thread_start_us":116,"threads_started":1}
I20260620 12:28:57.880149 18958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6602b6194ebb9951bb76ce30db0b" candidate_uuid: "bed459d353ef4bf5895d23e7bab6e697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27cada6e8dfe4736b40eb250d0646f47" is_pre_election: true
I20260620 12:28:57.880153 18824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6602b6194ebb9951bb76ce30db0b" candidate_uuid: "bed459d353ef4bf5895d23e7bab6e697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a86dd923c8084a8daf1702e641f03ff7" is_pre_election: true
I20260620 12:28:57.880434 18824 raft_consensus.cc:2454] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed459d353ef4bf5895d23e7bab6e697 in term 0.
I20260620 12:28:57.880434 18958 raft_consensus.cc:2454] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bed459d353ef4bf5895d23e7bab6e697 in term 0.
I20260620 12:28:57.880920 18626 leader_election.cc:302] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cada6e8dfe4736b40eb250d0646f47, bed459d353ef4bf5895d23e7bab6e697; no voters:
I20260620 12:28:57.881155 19027 raft_consensus.cc:2781] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:28:57.881238 19027 raft_consensus.cc:482] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:28:57.881283 19027 raft_consensus.cc:3037] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:57.882266 19027 raft_consensus.cc:504] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.882586 19027 leader_election.cc:288] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [CANDIDATE]: Term 1 election: Requested vote from peers 27cada6e8dfe4736b40eb250d0646f47 (127.0.98.3:35933), a86dd923c8084a8daf1702e641f03ff7 (127.0.98.2:44213)
I20260620 12:28:57.882979 18958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6602b6194ebb9951bb76ce30db0b" candidate_uuid: "bed459d353ef4bf5895d23e7bab6e697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27cada6e8dfe4736b40eb250d0646f47"
I20260620 12:28:57.883077 18824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f6602b6194ebb9951bb76ce30db0b" candidate_uuid: "bed459d353ef4bf5895d23e7bab6e697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a86dd923c8084a8daf1702e641f03ff7"
I20260620 12:28:57.883104 18958 raft_consensus.cc:3037] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:57.883188 18824 raft_consensus.cc:3037] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:57.884377 18824 raft_consensus.cc:2454] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed459d353ef4bf5895d23e7bab6e697 in term 1.
I20260620 12:28:57.884377 18958 raft_consensus.cc:2454] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed459d353ef4bf5895d23e7bab6e697 in term 1.
I20260620 12:28:57.884711 18626 leader_election.cc:302] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a86dd923c8084a8daf1702e641f03ff7, bed459d353ef4bf5895d23e7bab6e697; no voters:
I20260620 12:28:57.884949 19027 raft_consensus.cc:2781] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:57.885169 19027 raft_consensus.cc:686] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [term 1 LEADER]: Becoming Leader. State: Replica: bed459d353ef4bf5895d23e7bab6e697, State: Running, Role: LEADER
I20260620 12:28:57.885512 19027 consensus_queue.cc:237] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260620 12:28:57.888825 18545 catalog_manager.cc:5495] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 reported cstate change: term changed from 0 to 1, leader changed from <none> to bed459d353ef4bf5895d23e7bab6e697 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bed459d353ef4bf5895d23e7bab6e697" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed459d353ef4bf5895d23e7bab6e697" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } health_report { overall_health: UNKNOWN } } }
I20260620 12:28:57.907416 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.903536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.0.98.1:36177)
0620 12:28:57.903681 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:57.903686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:57.903700 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:28:57.903873 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:57.903875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:57.903930 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:57.904041 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:28:57.904321 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.904883 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.905820 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:57.906063 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:57.906177 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:57.906482 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:28:57.906960 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:28:57.907119 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:57.907219 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 12:28:57.907255 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:28:57.913125 18958 raft_consensus.cc:1261] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 1 FOLLOWER]: Refusing update from remote peer bed459d353ef4bf5895d23e7bab6e697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:57.913125 18824 raft_consensus.cc:1261] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 1 FOLLOWER]: Refusing update from remote peer bed459d353ef4bf5895d23e7bab6e697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:57.913918 19027 consensus_queue.cc:1035] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a86dd923c8084a8daf1702e641f03ff7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:57.914207 19031 consensus_queue.cc:1035] T bd9f6602b6194ebb9951bb76ce30db0b P bed459d353ef4bf5895d23e7bab6e697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27cada6e8dfe4736b40eb250d0646f47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:57.920913 19034 mvcc.cc:205] Tried to move back new op lower bound from 7298902171284873216 to 7298902171181903872. Current Snapshot: MvccSnapshot[applied={T|T < 7298902171284873216}]
I20260620 12:28:57.929557 18544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:57.929737 18544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:57.932184 18544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d11da2be444499c80d4eff2775af0e1: Sending DeleteTablet for 3 replicas of tablet bd9f6602b6194ebb9951bb76ce30db0b
I20260620 12:28:57.933097 18669 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f6602b6194ebb9951bb76ce30db0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44142
I20260620 12:28:57.933439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 18607
I20260620 12:28:57.933562 18938 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f6602b6194ebb9951bb76ce30db0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41740
I20260620 12:28:57.934190 18804 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f6602b6194ebb9951bb76ce30db0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37386
I20260620 12:28:57.934635 19046 tablet_replica.cc:330] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: stopping tablet replica
I20260620 12:28:57.935021 19046 raft_consensus.cc:2229] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:57.935032 19045 tablet_replica.cc:330] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: stopping tablet replica
I20260620 12:28:57.935350 19045 raft_consensus.cc:2229] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:57.935395 19046 raft_consensus.cc:2258] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:57.935652 19045 raft_consensus.cc:2258] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:57.936980 19046 ts_tablet_manager.cc:1905] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:57.937219 19045 ts_tablet_manager.cc:1905] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:57.939673 19045 ts_tablet_manager.cc:1918] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:57.939785 19045 log.cc:1196] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/bd9f6602b6194ebb9951bb76ce30db0b
I20260620 12:28:57.940081 19045 ts_tablet_manager.cc:1939] T bd9f6602b6194ebb9951bb76ce30db0b P 27cada6e8dfe4736b40eb250d0646f47: Deleting consensus metadata
I20260620 12:28:57.940402 19046 ts_tablet_manager.cc:1918] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:57.940511 19046 log.cc:1196] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/bd9f6602b6194ebb9951bb76ce30db0b
I20260620 12:28:57.940828 19046 ts_tablet_manager.cc:1939] T bd9f6602b6194ebb9951bb76ce30db0b P a86dd923c8084a8daf1702e641f03ff7: Deleting consensus metadata
I20260620 12:28:57.941270 18532 catalog_manager.cc:4841] TS 27cada6e8dfe4736b40eb250d0646f47 (127.0.98.3:35933): tablet bd9f6602b6194ebb9951bb76ce30db0b (table test-table [id=39af183d81ee4cbf9d6365abc6d69149]) successfully deleted
W20260620 12:28:57.942104 18531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36177 (error 108)
I20260620 12:28:57.942183 18532 catalog_manager.cc:4841] TS a86dd923c8084a8daf1702e641f03ff7 (127.0.98.2:44213): tablet bd9f6602b6194ebb9951bb76ce30db0b (table test-table [id=39af183d81ee4cbf9d6365abc6d69149]) successfully deleted
I20260620 12:28:57.942248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 18741
I20260620 12:28:57.943761 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:57.943416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36177 (local address 127.0.0.1:44154)
0620 12:28:57.943573 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:57.943652 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 12:28:57.943930 18531 catalog_manager.cc:4584] TS bed459d353ef4bf5895d23e7bab6e697 (127.0.98.1:36177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd9f6602b6194ebb9951bb76ce30db0b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36177: connect: Connection refused (error 111)
I20260620 12:28:57.948719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 18875
I20260620 12:28:57.954063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 18513
2026-06-20T12:28:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2242 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T12:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:28:57Z Disabled control of system clock
I20260620 12:28:57.986496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41129
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:28:58.095270 19051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:58.095505 19051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:58.095539 19051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:58.099436 19051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:28:58.099481 19051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:58.099496 19051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:58.099517 19051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:28:58.099532 19051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:28:58.104190 19051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:58.105059 19051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:58.106287 19051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:58.114393 19057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:58.114394 19056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:58.114401 19059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:58.114634 19051 server_base.cc:1034] running on GCE node
I20260620 12:28:58.115165 19051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:58.116204 19051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:58.117420 19051 hybrid_clock.cc:648] HybridClock initialized: now 1781958538117403 us; error 30 us; skew 500 ppm
I20260620 12:28:58.119295 19051 webserver.cc:460] Webserver started at http://127.0.98.62:38339/ using document root <none> and password file <none>
I20260620 12:28:58.119855 19051 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:58.119944 19051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:58.120172 19051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:58.122196 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "bdde163f0fae4d35811347aaaa26096c"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "5e66189ffe596f25e835fe4e6510db08"
server_key_iv: "2eb71e789b20d09e8f27f5635d6276bf"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.122819 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdde163f0fae4d35811347aaaa26096c"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "5e66189ffe596f25e835fe4e6510db08"
server_key_iv: "2eb71e789b20d09e8f27f5635d6276bf"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.126228 19051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:28:58.128595 19065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.129768 19051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:58.129902 19051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "bdde163f0fae4d35811347aaaa26096c"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "5e66189ffe596f25e835fe4e6510db08"
server_key_iv: "2eb71e789b20d09e8f27f5635d6276bf"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.130018 19051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:58.167544 19051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:58.168300 19051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:58.168504 19051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:58.175648 19051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41129
I20260620 12:28:58.175657 19117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41129 every 8 connection(s)
I20260620 12:28:58.176654 19051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:28:58.179831 19118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:58.182431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19051
I20260620 12:28:58.182590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:28:58.182897 392 external_mini_cluster.cc:1428] Setting key 744c32b5d473450fc21fd4644f3af122
I20260620 12:28:58.185642 19118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c: Bootstrap starting.
I20260620 12:28:58.187860 19118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:58.188570 19118 log.cc:824] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:58.190326 19118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c: No bootstrap required, opened a new log
I20260620 12:28:58.191854 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.184286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.0.98.62:41129)
0620 12:28:58.184914 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.184929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.184964 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:28:58.185566 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.185584 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.186019 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.186351 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:58.186820 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.187743 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.188675 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.189133 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.189307 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.190149 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:58.190154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:58.190166 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:58.190579 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:58.190633 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:58.190692 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:58.190807 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.190920 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.191199 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":131,"threads_started":1}
I20260620 12:28:58.193653 19118 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdde163f0fae4d35811347aaaa26096c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260620 12:28:58.193902 19118 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:58.193948 19118 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdde163f0fae4d35811347aaaa26096c, State: Initialized, Role: FOLLOWER
I20260620 12:28:58.194377 19118 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdde163f0fae4d35811347aaaa26096c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260620 12:28:58.194563 19118 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:28:58.194635 19118 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:28:58.194749 19118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:58.195583 19118 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdde163f0fae4d35811347aaaa26096c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260620 12:28:58.195914 19118 leader_election.cc:302] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdde163f0fae4d35811347aaaa26096c; no voters:
I20260620 12:28:58.196195 19118 leader_election.cc:288] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:28:58.196352 19123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:58.196578 19123 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [term 1 LEADER]: Becoming Leader. State: Replica: bdde163f0fae4d35811347aaaa26096c, State: Running, Role: LEADER
I20260620 12:28:58.197145 19123 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdde163f0fae4d35811347aaaa26096c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260620 12:28:58.197427 19118 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:28:58.198591 19125 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdde163f0fae4d35811347aaaa26096c. Latest consensus state: current_term: 1 leader_uuid: "bdde163f0fae4d35811347aaaa26096c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdde163f0fae4d35811347aaaa26096c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } } }
I20260620 12:28:58.198770 19125 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [sys.catalog]: This master's current role is: LEADER
I20260620 12:28:58.198835 19124 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdde163f0fae4d35811347aaaa26096c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdde163f0fae4d35811347aaaa26096c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } } }
I20260620 12:28:58.198956 19124 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c [sys.catalog]: This master's current role is: LEADER
W20260620 12:28:58.201687 19138 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:28:58.201752 19138 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:28:58.201845 19132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:28:58.202989 19132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:28:58.208526 19132 catalog_manager.cc:1347] Generated new cluster ID: 539c0541cbde4bb9912a7fbe69d65256
I20260620 12:28:58.208586 19132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:28:58.229094 19132 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:28:58.230091 19132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:28:58.235174 19132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c: Generated new TSK 0
I20260620 12:28:58.235936 19132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:28:58.242908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:28:58.352582 19142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:58.352814 19142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:58.352851 19142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:58.356750 19142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:58.356796 19142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:58.356875 19142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:28:58.361392 19142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:58.362383 19142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:58.363762 19142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:58.373837 19148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:58.373862 19150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:58.373838 19147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:58.373907 19142 server_base.cc:1034] running on GCE node
I20260620 12:28:58.374452 19142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:58.375137 19142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:58.376335 19142 hybrid_clock.cc:648] HybridClock initialized: now 1781958538376328 us; error 33 us; skew 500 ppm
I20260620 12:28:58.378157 19142 webserver.cc:460] Webserver started at http://127.0.98.1:34003/ using document root <none> and password file <none>
I20260620 12:28:58.378882 19142 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:58.378973 19142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:58.379180 19142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:58.381184 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "8012c0003b67595c53ca548a54a1cbc4"
server_key_iv: "2eafac3ad8498919ec327caa3a0f95fa"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.381778 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "8012c0003b67595c53ca548a54a1cbc4"
server_key_iv: "2eafac3ad8498919ec327caa3a0f95fa"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.385478 19142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 12:28:58.387950 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.389128 19142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:28:58.389259 19142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "8012c0003b67595c53ca548a54a1cbc4"
server_key_iv: "2eafac3ad8498919ec327caa3a0f95fa"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.389365 19142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:58.404052 19142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:58.404659 19142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:58.404857 19142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:58.405385 19142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:58.406252 19142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:58.406320 19142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.406389 19142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:58.406440 19142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.414893 19142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46877
I20260620 12:28:58.414933 19269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46877 every 8 connection(s)
I20260620 12:28:58.415760 19142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:28:58.419123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19142
I20260620 12:28:58.419268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:28:58.419561 392 external_mini_cluster.cc:1428] Setting key aa38ea2a114d737679e07ea07e8be1ee
I20260620 12:28:58.421586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:28:58.425820 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.417434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42297 (local address 127.0.98.62:41129)
0620 12:28:58.417565 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.417576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.418570 (+ 994us) server_negotiation.cc:407] Connection header received
0620 12:28:58.420126 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.420130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.420185 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.420276 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:58.421353 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.422185 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.423825 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.424056 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.424158 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.425321 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:58.425327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:58.425333 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:58.425346 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:58.425374 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:58.425382 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:58.425461 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.425615 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.425674 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:28:58.426585 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.417810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.1:42297)
0620 12:28:58.418428 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.418466 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.419887 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.420463 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.420471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.420584 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:58.421197 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.421212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.422325 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.422328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.423693 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.423706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.424232 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.424236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.424430 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.425057 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:28:58.425059 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:58.425096 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:28:58.425154 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:58.425465 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:58.425476 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:58.425483 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.425615 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.425869 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":106,"threads_started":1}
I20260620 12:28:58.427764 19270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260620 12:28:58.427999 19270 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:58.428463 19270 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260620 12:28:58.430101 19082 ts_manager.cc:194] Registered new tserver with Master: 90c32ac7e4fc4545a0f7f5b2e34a16af (127.0.98.1:46877)
I20260620 12:28:58.431833 19082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42297
W20260620 12:28:58.531458 19273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:58.531673 19273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:58.531708 19273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:58.535764 19273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:58.535812 19273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:58.535864 19273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:28:58.540441 19273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:58.541318 19273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:58.542515 19273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:58.551671 19281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:58.551762 19273 server_base.cc:1034] running on GCE node
W20260620 12:28:58.551647 19279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:58.551663 19278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:58.552248 19273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:58.552834 19273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:58.554023 19273 hybrid_clock.cc:648] HybridClock initialized: now 1781958538554012 us; error 25 us; skew 500 ppm
I20260620 12:28:58.555747 19273 webserver.cc:460] Webserver started at http://127.0.98.2:43473/ using document root <none> and password file <none>
I20260620 12:28:58.556282 19273 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:58.556371 19273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:58.556599 19273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:58.558509 19273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "d32a5ec829384069abbcbe9e9a22fd53"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "82e528178e6cbbdbdec5d5ad04fa3774"
server_key_iv: "6dec9a4cfedd1388f2f878cfc125742a"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.559121 19273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d32a5ec829384069abbcbe9e9a22fd53"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "82e528178e6cbbdbdec5d5ad04fa3774"
server_key_iv: "6dec9a4cfedd1388f2f878cfc125742a"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.562516 19273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:28:58.564843 19287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.565987 19273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:58.566116 19273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "d32a5ec829384069abbcbe9e9a22fd53"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "82e528178e6cbbdbdec5d5ad04fa3774"
server_key_iv: "6dec9a4cfedd1388f2f878cfc125742a"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.566221 19273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:58.584080 19273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:58.584697 19273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:58.584883 19273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:58.585423 19273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:58.586293 19273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:58.586365 19273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.586434 19273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:58.586494 19273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.595472 19273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43019
I20260620 12:28:58.595490 19400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43019 every 8 connection(s)
I20260620 12:28:58.596297 19273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:28:58.598001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19273
I20260620 12:28:58.598119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:28:58.598541 392 external_mini_cluster.cc:1428] Setting key a8cf023da44691f1f4efff872ed01d5e
I20260620 12:28:58.600808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--builtin_ntp_servers=127.0.98.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:28:58.606374 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.598161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49355 (local address 127.0.98.62:41129)
0620 12:28:58.598302 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.598307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.599562 (+ 1255us) server_negotiation.cc:407] Connection header received
0620 12:28:58.601321 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.601324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.601372 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.601440 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:58.602575 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.603178 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.604392 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.604633 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.604745 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.605868 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:58.605872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:58.605874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:58.605888 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:58.605909 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:58.605924 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:58.606004 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.606174 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.606233 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:28:58.607234 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.598558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.2:49355)
0620 12:28:58.599389 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.599441 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.601094 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.601612 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.601623 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.601734 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:58.602432 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.602449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.603306 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.603309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.604270 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.604280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.604782 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.604786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.604968 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.605610 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:28:58.605611 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:58.605650 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:28:58.605714 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:58.606006 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:58.606017 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:58.606024 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.606159 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.606423 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":112,"threads_started":1}
I20260620 12:28:58.608376 19401 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260620 12:28:58.608628 19401 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:58.609113 19401 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260620 12:28:58.610172 19082 ts_manager.cc:194] Registered new tserver with Master: d32a5ec829384069abbcbe9e9a22fd53 (127.0.98.2:43019)
I20260620 12:28:58.611006 19082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49355
W20260620 12:28:58.708658 19404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:28:58.708875 19404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:28:58.708909 19404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:28:58.712810 19404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:28:58.712852 19404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:28:58.712918 19404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:28:58.717460 19404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:28:58.718312 19404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:28:58.719599 19404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:28:58.729033 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:28:58.729099 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:58.729076 19404 server_base.cc:1034] running on GCE node
W20260620 12:28:58.728999 19412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:28:58.729574 19404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:28:58.730178 19404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:28:58.731334 19404 hybrid_clock.cc:648] HybridClock initialized: now 1781958538731302 us; error 40 us; skew 500 ppm
I20260620 12:28:58.733178 19404 webserver.cc:460] Webserver started at http://127.0.98.3:33807/ using document root <none> and password file <none>
I20260620 12:28:58.733800 19404 fs_manager.cc:359] Metadata directory not provided
I20260620 12:28:58.733865 19404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:28:58.734033 19404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:28:58.736188 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2681c3d46804a5dabe6223ab4258a49"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "ade44f4a21c8c1ae0c77312c7395463d"
server_key_iv: "a596b1c4a4094abc6e3ff7ca3632405a"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.736668 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2681c3d46804a5dabe6223ab4258a49"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "ade44f4a21c8c1ae0c77312c7395463d"
server_key_iv: "a596b1c4a4094abc6e3ff7ca3632405a"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.740146 19404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:28:58.742401 19418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.743595 19404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:28:58.743690 19404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "c2681c3d46804a5dabe6223ab4258a49"
format_stamp: "Formatted at 2026-06-20 12:28:58 on dist-test-slave-bwpp"
server_key: "ade44f4a21c8c1ae0c77312c7395463d"
server_key_iv: "a596b1c4a4094abc6e3ff7ca3632405a"
server_key_version: "encryptionkey@0"
I20260620 12:28:58.743768 19404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:28:58.767012 19404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:28:58.767656 19404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:28:58.767800 19404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:28:58.768405 19404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:28:58.769382 19404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:28:58.769431 19404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.769472 19404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:28:58.769488 19404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:28:58.779069 19404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46227
I20260620 12:28:58.779094 19531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46227 every 8 connection(s)
I20260620 12:28:58.780001 19404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:28:58.786839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19404
I20260620 12:28:58.786973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:28:58.787294 392 external_mini_cluster.cc:1428] Setting key 87ce65600be2eb84265d1b0659bf6c17
I20260620 12:28:58.789618 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.782028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52399 (local address 127.0.98.62:41129)
0620 12:28:58.782201 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.782206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.783380 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 12:28:58.784710 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.784713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.784763 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.784846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:58.785691 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.786234 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.787564 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.787899 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.788018 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.789115 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:58.789119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:58.789122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:58.789135 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:58.789157 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:58.789173 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:58.789261 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.789452 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.789505 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:28:58.790525 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.782505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.3:52399)
0620 12:28:58.783230 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.783270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.784506 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.784979 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.784986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.785067 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:28:58.785551 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.785563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.786344 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.786346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.787411 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.787424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.788056 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.788060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.788226 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.788854 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:28:58.788855 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:28:58.788896 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:28:58.788963 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:28:58.789274 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:28:58.789281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:28:58.789286 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.789432 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.789699 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":188,"threads_started":1}
I20260620 12:28:58.791639 19532 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260620 12:28:58.791879 19532 heartbeater.cc:464] Registering TS with master...
I20260620 12:28:58.792419 19532 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260620 12:28:58.793447 19082 ts_manager.cc:194] Registered new tserver with Master: c2681c3d46804a5dabe6223ab4258a49 (127.0.98.3:46227)
I20260620 12:28:58.794061 19082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52399
I20260620 12:28:58.801105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:28:58.806594 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.803107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.98.62:41129)
0620 12:28:58.803255 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.803262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.803395 (+ 133us) server_negotiation.cc:407] Connection header received
0620 12:28:58.803493 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.803496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.803535 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.803595 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:28:58.803809 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.804298 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.805078 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.805278 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.805351 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.806090 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:28:58.806093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:28:58.806095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:28:58.806106 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:28:58.806123 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:28:58.806130 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:28:58.806228 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.806361 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.806418 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:28:58.809434 19082 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:28:58.812520 19082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:28:58.826885 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.821430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43019 (local address 127.0.0.1:59764)
0620 12:28:58.822141 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.822156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.822237 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.822865 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.822870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.822892 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:58.823041 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.823048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.824535 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.824538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.825499 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.825508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.826307 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.826311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.826426 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.826464 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.826576 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.826711 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":53,"thread_start_us":78,"threads_started":1}
I20260620 12:28:58.827105 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.821198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.98.1:46877)
0620 12:28:58.822511 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.822522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.822543 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:28:58.822607 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.822611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.822800 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.822922 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:58.823330 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.824136 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.825551 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.826360 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.826461 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.826633 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.826843 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.826949 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1212,"thread_start_us":132,"threads_started":1}
I20260620 12:28:58.827417 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.821061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46877 (local address 127.0.0.1:38706)
0620 12:28:58.821750 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.821801 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.821946 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.823037 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.823043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.823064 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:58.823183 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.823188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.824268 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.824271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.825402 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.825419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.826579 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.826584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.826698 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.826718 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.826792 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.826845 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":37,"thread_start_us":83,"threads_started":1}
I20260620 12:28:58.828361 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.821798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46227 (local address 127.0.0.1:56438)
0620 12:28:58.822293 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.822309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.822397 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.823071 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.823074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.823086 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:58.823182 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.823188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.824535 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.824538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.826591 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.826601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.827958 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.827962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.828042 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.828064 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.828140 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.828224 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":144,"thread_start_us":85,"threads_started":1}
I20260620 12:28:58.828594 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.821871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.0.98.2:43019)
0620 12:28:58.822343 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.822357 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.822381 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:28:58.822464 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.822470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.822703 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.822870 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:58.823170 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.824405 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.825631 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.826177 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.826309 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.826504 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.826969 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.828427 (+ 1458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":110,"threads_started":1}
I20260620 12:28:58.828861 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.822182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.0.98.3:46227)
0620 12:28:58.822565 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.822574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.822592 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:28:58.822697 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.822704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.822911 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.823055 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:58.823297 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.824405 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.826761 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.827596 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.827689 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.827891 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.828619 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.828711 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":94,"threads_started":1}
I20260620 12:28:58.831513 19204 tablet_service.cc:1467] Processing CreateTablet for tablet 0b39984be9a04f9e811fb8191128e578 (DEFAULT_TABLE table=test-table [id=d33eb3398e3146c596a496be69e60643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:58.832028 19335 tablet_service.cc:1467] Processing CreateTablet for tablet 0b39984be9a04f9e811fb8191128e578 (DEFAULT_TABLE table=test-table [id=d33eb3398e3146c596a496be69e60643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:58.832921 19204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b39984be9a04f9e811fb8191128e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:58.833096 19335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b39984be9a04f9e811fb8191128e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:58.832962 19466 tablet_service.cc:1467] Processing CreateTablet for tablet 0b39984be9a04f9e811fb8191128e578 (DEFAULT_TABLE table=test-table [id=d33eb3398e3146c596a496be69e60643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:28:58.833967 19466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b39984be9a04f9e811fb8191128e578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:28:58.841094 19551 tablet_bootstrap.cc:492] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Bootstrap starting.
I20260620 12:28:58.841920 19549 tablet_bootstrap.cc:492] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Bootstrap starting.
I20260620 12:28:58.842983 19550 tablet_bootstrap.cc:492] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af: Bootstrap starting.
I20260620 12:28:58.843151 19551 tablet_bootstrap.cc:654] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:58.843828 19551 log.cc:824] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:58.843906 19549 tablet_bootstrap.cc:654] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:58.844730 19549 log.cc:824] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:58.844980 19550 tablet_bootstrap.cc:654] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af: Neither blocks nor log segments found. Creating new log.
I20260620 12:28:58.845566 19550 log.cc:824] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af: Log is configured to *not* fsync() on all Append() calls
I20260620 12:28:58.845781 19551 tablet_bootstrap.cc:492] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: No bootstrap required, opened a new log
I20260620 12:28:58.845971 19551 ts_tablet_manager.cc:1397] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:28:58.847265 19549 tablet_bootstrap.cc:492] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: No bootstrap required, opened a new log
I20260620 12:28:58.847445 19549 ts_tablet_manager.cc:1397] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:28:58.849442 19551 raft_consensus.cc:348] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.849670 19551 raft_consensus.cc:374] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:58.849725 19551 raft_consensus.cc:729] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2681c3d46804a5dabe6223ab4258a49, State: Initialized, Role: FOLLOWER
I20260620 12:28:58.850113 19551 consensus_queue.cc:260] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.850277 19549 raft_consensus.cc:348] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.850553 19549 raft_consensus.cc:374] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:58.850613 19549 raft_consensus.cc:729] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32a5ec829384069abbcbe9e9a22fd53, State: Initialized, Role: FOLLOWER
I20260620 12:28:58.850787 19532 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
I20260620 12:28:58.851039 19551 ts_tablet_manager.cc:1428] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:28:58.851045 19549 consensus_queue.cc:260] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.851605 19401 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
I20260620 12:28:58.852873 19550 tablet_bootstrap.cc:492] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af: No bootstrap required, opened a new log
I20260620 12:28:58.853053 19550 ts_tablet_manager.cc:1397] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 12:28:58.853143 19549 ts_tablet_manager.cc:1428] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:28:58.856861 19550 raft_consensus.cc:348] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.857072 19550 raft_consensus.cc:374] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:28:58.857105 19550 raft_consensus.cc:729] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c32ac7e4fc4545a0f7f5b2e34a16af, State: Initialized, Role: FOLLOWER
I20260620 12:28:58.857508 19550 consensus_queue.cc:260] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.858088 19270 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
I20260620 12:28:58.858367 19550 ts_tablet_manager.cc:1428] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:28:58.861690 19557 raft_consensus.cc:482] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:28:58.861905 19557 raft_consensus.cc:504] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.863230 19557 leader_election.cc:288] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2681c3d46804a5dabe6223ab4258a49 (127.0.98.3:46227), d32a5ec829384069abbcbe9e9a22fd53 (127.0.98.2:43019)
I20260620 12:28:58.867321 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.863554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50049 (local address 127.0.98.3:46227)
0620 12:28:58.863665 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.863672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.863690 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:28:58.863791 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.863794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.863849 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.863915 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:58.864181 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.864901 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.866167 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.866776 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.866856 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.866902 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.867136 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.867189 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 12:28:58.867348 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.863460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46227 (local address 127.0.98.1:50049)
0620 12:28:58.863577 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.863595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.863686 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.863940 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.863942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.863957 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:58.864059 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.864064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.865007 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.865011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.866042 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.866051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.866888 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.866892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.866998 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.867025 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.867105 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.867177 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260620 12:28:58.867889 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.863542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38875 (local address 127.0.98.2:43019)
0620 12:28:58.863666 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.863671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.864251 (+ 580us) server_negotiation.cc:407] Connection header received
0620 12:28:58.864328 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.864331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.864371 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.864434 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:28:58.864706 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.865429 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.866568 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.867193 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.867298 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.867354 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.867729 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.867800 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:28:58.867893 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.863463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43019 (local address 127.0.98.1:38875)
0620 12:28:58.864146 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.864161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:28:58.864254 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:28:58.864468 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:28:58.864471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:28:58.864487 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:28:58.864600 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.864606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.865545 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.865548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.866438 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:28:58.866448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.867318 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.867322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:28:58.867593 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.867618 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:28:58.867693 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:28:58.867762 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":145,"thread_start_us":101,"threads_started":1}
I20260620 12:28:58.867928 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b39984be9a04f9e811fb8191128e578" candidate_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2681c3d46804a5dabe6223ab4258a49" is_pre_election: true
I20260620 12:28:58.868201 19486 raft_consensus.cc:2454] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c32ac7e4fc4545a0f7f5b2e34a16af in term 0.
I20260620 12:28:58.868635 19355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b39984be9a04f9e811fb8191128e578" candidate_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32a5ec829384069abbcbe9e9a22fd53" is_pre_election: true
I20260620 12:28:58.868726 19159 leader_election.cc:302] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c32ac7e4fc4545a0f7f5b2e34a16af, c2681c3d46804a5dabe6223ab4258a49; no voters:
I20260620 12:28:58.868927 19355 raft_consensus.cc:2454] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c32ac7e4fc4545a0f7f5b2e34a16af in term 0.
I20260620 12:28:58.869014 19557 raft_consensus.cc:2781] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:28:58.869103 19557 raft_consensus.cc:482] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:28:58.869160 19557 raft_consensus.cc:3037] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:58.870235 19557 raft_consensus.cc:504] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.870623 19557 leader_election.cc:288] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [CANDIDATE]: Term 1 election: Requested vote from peers c2681c3d46804a5dabe6223ab4258a49 (127.0.98.3:46227), d32a5ec829384069abbcbe9e9a22fd53 (127.0.98.2:43019)
I20260620 12:28:58.870913 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b39984be9a04f9e811fb8191128e578" candidate_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2681c3d46804a5dabe6223ab4258a49"
I20260620 12:28:58.871037 19355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b39984be9a04f9e811fb8191128e578" candidate_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32a5ec829384069abbcbe9e9a22fd53"
I20260620 12:28:58.871062 19486 raft_consensus.cc:3037] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:58.871162 19355 raft_consensus.cc:3037] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:28:58.872454 19355 raft_consensus.cc:2454] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c32ac7e4fc4545a0f7f5b2e34a16af in term 1.
I20260620 12:28:58.872443 19486 raft_consensus.cc:2454] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c32ac7e4fc4545a0f7f5b2e34a16af in term 1.
I20260620 12:28:58.872762 19159 leader_election.cc:302] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c32ac7e4fc4545a0f7f5b2e34a16af, c2681c3d46804a5dabe6223ab4258a49; no voters:
I20260620 12:28:58.872936 19557 raft_consensus.cc:2781] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:28:58.873178 19557 raft_consensus.cc:686] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [term 1 LEADER]: Becoming Leader. State: Replica: 90c32ac7e4fc4545a0f7f5b2e34a16af, State: Running, Role: LEADER
I20260620 12:28:58.873482 19557 consensus_queue.cc:237] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } }
I20260620 12:28:58.876652 19081 catalog_manager.cc:5495] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af reported cstate change: term changed from 0 to 1, leader changed from <none> to 90c32ac7e4fc4545a0f7f5b2e34a16af (127.0.98.1). New cstate: current_term: 1 leader_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c32ac7e4fc4545a0f7f5b2e34a16af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 } health_report { overall_health: UNKNOWN } } }
I20260620 12:28:58.894100 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.890393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.98.1:46877)
0620 12:28:58.890536 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:28:58.890540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:28:58.890553 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:28:58.890687 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:28:58.890689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:28:58.890751 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:28:58.890835 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:28:58.891062 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.891538 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.892505 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:28:58.892855 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:28:58.892935 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:28:58.893236 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:28:58.893674 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:28:58.893842 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 12:28:58.893955 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 12:28:58.893986 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:28:58.900007 19486 raft_consensus.cc:1261] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 1 FOLLOWER]: Refusing update from remote peer 90c32ac7e4fc4545a0f7f5b2e34a16af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:58.900007 19355 raft_consensus.cc:1261] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 1 FOLLOWER]: Refusing update from remote peer 90c32ac7e4fc4545a0f7f5b2e34a16af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:28:58.900722 19557 consensus_queue.cc:1035] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32a5ec829384069abbcbe9e9a22fd53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:58.900934 19559 consensus_queue.cc:1035] T 0b39984be9a04f9e811fb8191128e578 P 90c32ac7e4fc4545a0f7f5b2e34a16af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2681c3d46804a5dabe6223ab4258a49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:28:58.908676 19563 mvcc.cc:205] Tried to move back new op lower bound from 7298902175326957568 to 7298902175228510208. Current Snapshot: MvccSnapshot[applied={T|T < 7298902175326957568}]
I20260620 12:28:58.910543 19564 mvcc.cc:205] Tried to move back new op lower bound from 7298902175326957568 to 7298902175228510208. Current Snapshot: MvccSnapshot[applied={T|T < 7298902175326957568}]
I20260620 12:28:58.916519 19081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:58.916697 19081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:28:58.919118 19081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdde163f0fae4d35811347aaaa26096c: Sending DeleteTablet for 3 replicas of tablet 0b39984be9a04f9e811fb8191128e578
I20260620 12:28:58.919970 19466 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b39984be9a04f9e811fb8191128e578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:58 UTC) from {username='slave'} at 127.0.0.1:56438
I20260620 12:28:58.920013 19204 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b39984be9a04f9e811fb8191128e578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:58 UTC) from {username='slave'} at 127.0.0.1:38706
I20260620 12:28:58.920512 19335 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b39984be9a04f9e811fb8191128e578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:28:58 UTC) from {username='slave'} at 127.0.0.1:59764
I20260620 12:28:58.920662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19142
I20260620 12:28:58.921684 19573 tablet_replica.cc:330] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: stopping tablet replica
I20260620 12:28:58.922062 19573 raft_consensus.cc:2229] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:58.922492 19573 raft_consensus.cc:2258] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:58.922703 19574 tablet_replica.cc:330] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: stopping tablet replica
I20260620 12:28:58.923046 19574 raft_consensus.cc:2229] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:28:58.923341 19574 raft_consensus.cc:2258] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:28:58.924120 19573 ts_tablet_manager.cc:1905] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:58.924588 19574 ts_tablet_manager.cc:1905] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:28:58.926906 19574 ts_tablet_manager.cc:1918] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:58.927024 19574 log.cc:1196] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/0b39984be9a04f9e811fb8191128e578
I20260620 12:28:58.927299 19574 ts_tablet_manager.cc:1939] T 0b39984be9a04f9e811fb8191128e578 P d32a5ec829384069abbcbe9e9a22fd53: Deleting consensus metadata
I20260620 12:28:58.927874 19573 ts_tablet_manager.cc:1918] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:28:58.927981 19573 log.cc:1196] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/0b39984be9a04f9e811fb8191128e578
I20260620 12:28:58.928336 19573 ts_tablet_manager.cc:1939] T 0b39984be9a04f9e811fb8191128e578 P c2681c3d46804a5dabe6223ab4258a49: Deleting consensus metadata
I20260620 12:28:58.928476 19066 catalog_manager.cc:4841] TS d32a5ec829384069abbcbe9e9a22fd53 (127.0.98.2:43019): tablet 0b39984be9a04f9e811fb8191128e578 (table test-table [id=d33eb3398e3146c596a496be69e60643]) successfully deleted
W20260620 12:28:58.928982 19068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46877 (error 108)
I20260620 12:28:58.929246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19273
I20260620 12:28:58.929529 19068 catalog_manager.cc:4841] TS c2681c3d46804a5dabe6223ab4258a49 (127.0.98.3:46227): tablet 0b39984be9a04f9e811fb8191128e578 (table test-table [id=d33eb3398e3146c596a496be69e60643]) successfully deleted
I20260620 12:28:58.930462 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:28:58.930080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46877 (local address 127.0.0.1:38726)
0620 12:28:58.930265 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 12:28:58.930356 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 12:28:58.930619 19068 catalog_manager.cc:4584] TS 90c32ac7e4fc4545a0f7f5b2e34a16af (127.0.98.1:46877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b39984be9a04f9e811fb8191128e578: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46877: connect: Connection refused (error 111)
I20260620 12:28:58.935791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19404
I20260620 12:28:58.941385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19051
2026-06-20T12:28:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (985 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:28:58 dist-test-slave-bwpp krb5kdc[19578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:28:58 dist-test-slave-bwpp krb5kdc[19578](info): set up 2 sockets
Jun 20 12:28:58 dist-test-slave-bwpp krb5kdc[19578](info): commencing operation
krb5kdc: starting...
W20260620 12:28:59.974792 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:00.093504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43519
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:00.205310 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:00.205565 19591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:00.205605 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:00.209532 19591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:00.209584 19591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:00.209604 19591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:00.209662 19591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:00.209692 19591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:00.214825 19591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:00.215791 19591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:00.216765 19591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:00.224828 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:00.224885 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:00.224828 19597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:00.225131 19591 server_base.cc:1034] running on GCE node
I20260620 12:29:00.225674 19591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:00.226734 19591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:00.227928 19591 hybrid_clock.cc:648] HybridClock initialized: now 1781958540227913 us; error 35 us; skew 500 ppm
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:00.230538 19591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:00.231348 19591 webserver.cc:460] Webserver started at http://127.0.98.62:40135/ using document root <none> and password file <none>
I20260620 12:29:00.231875 19591 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:00.231961 19591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:00.232179 19591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:00.234042 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "0527fd757be441c39df7ae7106ade622"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "0c185c0e78fe8a273456f088e1cbcaf6"
server_key_iv: "15e1da0bc74cdc830c3c00229529196d"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.234581 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "0527fd757be441c39df7ae7106ade622"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "0c185c0e78fe8a273456f088e1cbcaf6"
server_key_iv: "15e1da0bc74cdc830c3c00229529196d"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.238075 19591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:29:00.240454 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.241644 19591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:00.241776 19591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "0527fd757be441c39df7ae7106ade622"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "0c185c0e78fe8a273456f088e1cbcaf6"
server_key_iv: "15e1da0bc74cdc830c3c00229529196d"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.241884 19591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:00.256080 19591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:00.258797 19591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:00.258971 19591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:00.265723 19591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43519
I20260620 12:29:00.265736 19658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43519 every 8 connection(s)
I20260620 12:29:00.266628 19591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:00.268833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19591
I20260620 12:29:00.268983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:00.269294 392 external_mini_cluster.cc:1428] Setting key 2632762452d4a00d1e7cdaa2cbe1e0dc
I20260620 12:29:00.269585 19659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:00.276684 19659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622: Bootstrap starting.
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:00.279031 19659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:00.279913 19659 log.cc:824] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:00.282130 19659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622: No bootstrap required, opened a new log
I20260620 12:29:00.284696 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.270698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.98.62:43519)
0620 12:29:00.271128 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:00.271148 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:00.271189 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:29:00.271908 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:00.271930 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:00.272350 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:00.272713 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:00.273390 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.274900 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.275855 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.276516 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.276676 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.279419 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:00.279425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:00.279441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:00.279470 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:00.281520 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.281951 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.281956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.281962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.282095 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.282446 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.282450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.282453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.282961 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:00.283140 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:00.283385 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 12:29:00.283672 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260620 12:29:00.286475 19659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0527fd757be441c39df7ae7106ade622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43519 } }
I20260620 12:29:00.286751 19659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:00.286823 19659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0527fd757be441c39df7ae7106ade622, State: Initialized, Role: FOLLOWER
I20260620 12:29:00.287238 19659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0527fd757be441c39df7ae7106ade622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43519 } }
I20260620 12:29:00.287359 19659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:00.287425 19659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:00.287528 19659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:00.288719 19659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0527fd757be441c39df7ae7106ade622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43519 } }
I20260620 12:29:00.289080 19659 leader_election.cc:302] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0527fd757be441c39df7ae7106ade622; no voters:
I20260620 12:29:00.289373 19659 leader_election.cc:288] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:00.289458 19664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:00.289753 19664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [term 1 LEADER]: Becoming Leader. State: Replica: 0527fd757be441c39df7ae7106ade622, State: Running, Role: LEADER
I20260620 12:29:00.290077 19664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0527fd757be441c39df7ae7106ade622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43519 } }
I20260620 12:29:00.290601 19659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:00.292327 19666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0527fd757be441c39df7ae7106ade622. Latest consensus state: current_term: 1 leader_uuid: "0527fd757be441c39df7ae7106ade622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0527fd757be441c39df7ae7106ade622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43519 } } }
I20260620 12:29:00.292450 19666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:00.292296 19665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0527fd757be441c39df7ae7106ade622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0527fd757be441c39df7ae7106ade622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43519 } } }
I20260620 12:29:00.292882 19665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:00.293174 19679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:00.295949 19679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:00.301019 19679 catalog_manager.cc:1347] Generated new cluster ID: 432d2424081e41a9a0d41ec921fa45c2
I20260620 12:29:00.301080 19679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:00.316512 19679 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:00.317472 19679 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:00.324280 19679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622: Generated new TSK 0
I20260620 12:29:00.325114 19679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:29:00.357712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43519
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:00.470526 19685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:00.470800 19685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:00.470845 19685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:00.474960 19685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:00.475018 19685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:00.475075 19685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:29:00.480309 19685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:00.481279 19685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:00.482252 19685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:00.491747 19693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:00.491876 19690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:00.491974 19685 server_base.cc:1034] running on GCE node
W20260620 12:29:00.491740 19691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:00.492484 19685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:00.493198 19685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:00.494379 19685 hybrid_clock.cc:648] HybridClock initialized: now 1781958540494350 us; error 41 us; skew 500 ppm
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:00.497187 19685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:29:00.497990 19685 webserver.cc:460] Webserver started at http://127.0.98.1:40409/ using document root <none> and password file <none>
I20260620 12:29:00.498538 19685 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:00.498595 19685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:00.498822 19685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:00.500733 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ed160bc77964eaeae016d5908d7228d"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "f07d43124aa7757a2a390a4d8bdc95bb"
server_key_iv: "eb3c6b4f9d382675d202ecb06f9e8533"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.501227 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ed160bc77964eaeae016d5908d7228d"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "f07d43124aa7757a2a390a4d8bdc95bb"
server_key_iv: "eb3c6b4f9d382675d202ecb06f9e8533"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.504796 19685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:29:00.507107 19700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.508504 19685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:00.508635 19685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "0ed160bc77964eaeae016d5908d7228d"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "f07d43124aa7757a2a390a4d8bdc95bb"
server_key_iv: "eb3c6b4f9d382675d202ecb06f9e8533"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.508742 19685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:00.521435 19685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:00.524150 19685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:00.524372 19685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:00.524945 19685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:29:00.525736 19685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:29:00.525784 19685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.525853 19685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:29:00.525884 19685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.534840 19685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35373
I20260620 12:29:00.534904 19813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35373 every 8 connection(s)
I20260620 12:29:00.535729 19685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:29:00.544054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19685
I20260620 12:29:00.544154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:29:00.544413 392 external_mini_cluster.cc:1428] Setting key da576938608d5f5000132067a1f6bf91
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:00.550451 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.537564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52439 (local address 127.0.98.62:43519)
0620 12:29:00.537697 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:00.537701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:00.538480 (+ 779us) server_negotiation.cc:407] Connection header received
0620 12:29:00.539823 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:00.539826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:00.539885 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:00.539994 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:00.541131 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.541671 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.542855 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.543093 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.543208 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.546358 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:00.546364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:00.546368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:00.546399 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:00.548093 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.548709 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.548711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.548712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.548786 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.549175 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.549180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.549185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.549430 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:00.549544 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:00.550042 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 12:29:00.550189 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:29:00.551170 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.537899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43519 (local address 127.0.98.1:52439)
0620 12:29:00.538327 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:00.538369 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:00.539614 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:00.540184 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:00.540191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:00.540529 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:00.540986 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:00.540999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.541802 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.541810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:00.542736 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:00.542746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.543246 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.543250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:00.543418 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.544317 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:00.544333 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:00.546187 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:00.548209 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:00.548216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:00.548232 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:00.548600 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:00.548874 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:00.548878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:00.548881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:00.549068 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:00.549544 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:00.549548 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:00.549791 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 12:29:00.549958 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 12:29:00.550199 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260620 12:29:00.552495 19814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43519
I20260620 12:29:00.552743 19814 heartbeater.cc:464] Registering TS with master...
I20260620 12:29:00.553208 19814 heartbeater.cc:511] Master 127.0.98.62:43519 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:29:00.554852 19623 ts_manager.cc:194] Registered new tserver with Master: 0ed160bc77964eaeae016d5908d7228d (127.0.98.1:35373)
I20260620 12:29:00.555941 19623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:29:00.573271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43519
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:00.682751 19819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:00.682996 19819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:00.683072 19819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:00.687134 19819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:00.687187 19819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:00.687250 19819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:29:00.692911 19819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:00.693889 19819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:00.694875 19819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:00.704308 19824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:00.704246 19825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:00.704238 19827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:00.704610 19819 server_base.cc:1034] running on GCE node
I20260620 12:29:00.705016 19819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:00.705638 19819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:00.706753 19819 hybrid_clock.cc:648] HybridClock initialized: now 1781958540706735 us; error 28 us; skew 500 ppm
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:00.709450 19819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:29:00.710227 19819 webserver.cc:460] Webserver started at http://127.0.98.2:35785/ using document root <none> and password file <none>
I20260620 12:29:00.710795 19819 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:00.710865 19819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:00.711030 19819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:00.713029 19819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "e526d0a743554aeb90b3a5c4f2c55aba"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "902cb63e4431a7c9216db249cbb360b0"
server_key_iv: "d147df35faa3b6b6f5448b1c93a2b1e1"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.713572 19819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e526d0a743554aeb90b3a5c4f2c55aba"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "902cb63e4431a7c9216db249cbb360b0"
server_key_iv: "d147df35faa3b6b6f5448b1c93a2b1e1"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.717085 19819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:29:00.719420 19834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.720625 19819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:00.720754 19819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "e526d0a743554aeb90b3a5c4f2c55aba"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "902cb63e4431a7c9216db249cbb360b0"
server_key_iv: "d147df35faa3b6b6f5448b1c93a2b1e1"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.720860 19819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:00.737411 19819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:00.740514 19819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:00.740725 19819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:00.741283 19819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:29:00.742075 19819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:29:00.742121 19819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.742188 19819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:29:00.742224 19819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.750948 19819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34803
I20260620 12:29:00.751027 19947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34803 every 8 connection(s)
I20260620 12:29:00.751834 19819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:29:00.759455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19819
I20260620 12:29:00.759593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:29:00.759899 392 external_mini_cluster.cc:1428] Setting key ba069c146e1b8de30b479863e1994a9a
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:00.766160 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.753595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48919 (local address 127.0.98.62:43519)
0620 12:29:00.753740 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:00.753745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:00.754624 (+ 879us) server_negotiation.cc:407] Connection header received
0620 12:29:00.755971 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:00.755974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:00.756023 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:00.756097 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:00.757196 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.757706 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.758896 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.759117 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.759222 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.762087 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:00.762093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:00.762099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:00.762133 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:00.763778 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.764439 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.764442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.764444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.764517 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.764906 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.764908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.764909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.765083 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:00.765158 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:00.765830 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 12:29:00.765997 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:29:00.767401 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.754036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43519 (local address 127.0.98.2:48919)
0620 12:29:00.754471 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:00.754512 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:00.755773 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:00.756249 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:00.756255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:00.756618 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:00.757078 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:00.757091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.757827 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.757830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:00.758791 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:00.758800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.759256 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.759258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:00.759426 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.760141 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:00.760163 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:00.761924 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:00.763895 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:00.763905 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:00.763923 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:00.764336 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:00.764607 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:00.764613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:00.764617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:00.764814 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:00.765187 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:00.765196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:00.765526 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 12:29:00.765756 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 12:29:00.766184 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260620 12:29:00.769275 19948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43519
I20260620 12:29:00.769536 19948 heartbeater.cc:464] Registering TS with master...
I20260620 12:29:00.770195 19948 heartbeater.cc:511] Master 127.0.98.62:43519 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:29:00.771325 19623 ts_manager.cc:194] Registered new tserver with Master: e526d0a743554aeb90b3a5c4f2c55aba (127.0.98.2:34803)
I20260620 12:29:00.771914 19623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:29:00.788792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43519
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:00.898344 19953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:00.898684 19953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:00.898767 19953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:00.903299 19953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:00.903383 19953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:00.903470 19953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:29:00.908732 19953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:00.909776 19953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:00.910861 19953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:00.920360 19959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:00.920382 19958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:00.920360 19961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:00.920477 19953 server_base.cc:1034] running on GCE node
I20260620 12:29:00.920992 19953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:00.921671 19953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:00.922874 19953 hybrid_clock.cc:648] HybridClock initialized: now 1781958540922859 us; error 28 us; skew 500 ppm
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:00.925841 19953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:29:00.926798 19953 webserver.cc:460] Webserver started at http://127.0.98.3:37403/ using document root <none> and password file <none>
I20260620 12:29:00.927357 19953 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:00.927449 19953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:00.927675 19953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:00.929816 19953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "00e3a96f35304c7c9d4c000b962641d7"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "271ffc0eafef0ae151adf1fe38a66bdb"
server_key_iv: "b0ed86291bce8195b72db1a475006f0f"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.930413 19953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "00e3a96f35304c7c9d4c000b962641d7"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "271ffc0eafef0ae151adf1fe38a66bdb"
server_key_iv: "b0ed86291bce8195b72db1a475006f0f"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.933987 19953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:29:00.936506 19968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.937753 19953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:29:00.937897 19953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "00e3a96f35304c7c9d4c000b962641d7"
format_stamp: "Formatted at 2026-06-20 12:29:00 on dist-test-slave-bwpp"
server_key: "271ffc0eafef0ae151adf1fe38a66bdb"
server_key_iv: "b0ed86291bce8195b72db1a475006f0f"
server_key_version: "encryptionkey@0"
I20260620 12:29:00.938015 19953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:00.949586 19953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:00.952769 19953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:00.952991 19953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:00.953598 19953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:29:00.954447 19953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:29:00.954519 19953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.954589 19953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:29:00.954633 19953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:00.963790 19953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35601
I20260620 12:29:00.963811 20081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35601 every 8 connection(s)
I20260620 12:29:00.964699 19953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 12:29:00 dist-test-slave-bwpp krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:00.974911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 19953
I20260620 12:29:00.975055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:29:00.975412 392 external_mini_cluster.cc:1428] Setting key 0d35d62485c520cb7b87dbd4128c41f1
I20260620 12:29:00.978574 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.966572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44917 (local address 127.0.98.62:43519)
0620 12:29:00.966738 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:00.966742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:00.967570 (+ 828us) server_negotiation.cc:407] Connection header received
0620 12:29:00.968894 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:00.968897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:00.968960 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:00.969071 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:00.970281 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.970878 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.972019 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.972248 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.972359 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.974772 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:00.974775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:00.974777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:00.974802 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:00.976556 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.977140 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.977143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.977145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.977234 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.977545 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.977548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.977550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.977772 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:00.977884 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:00.978357 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 12:29:00.978452 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:29:00.979425 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.966975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43519 (local address 127.0.98.3:44917)
0620 12:29:00.967415 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:00.967459 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:00.968674 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:00.969249 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:00.969255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:00.969617 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:00.970122 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:00.970135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.971011 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.971015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:00.971881 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:00.971890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.972397 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.972402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:00.972570 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.973118 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:00.973134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:00.974523 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:00.976687 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:00.976691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:00.976720 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:00.977051 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:00.977327 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:00.977329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:00.977330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:00.977456 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:00.977897 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:00.977902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:00.978134 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 12:29:00.978301 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 12:29:00.978547 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
I20260620 12:29:00.980553 20082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43519
I20260620 12:29:00.980793 20082 heartbeater.cc:464] Registering TS with master...
I20260620 12:29:00.981281 20082 heartbeater.cc:511] Master 127.0.98.62:43519 requested a full tablet report, sending...
I20260620 12:29:00.982364 19623 ts_manager.cc:194] Registered new tserver with Master: 00e3a96f35304c7c9d4c000b962641d7 (127.0.98.3:35601)
I20260620 12:29:00.982995 19623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44917
I20260620 12:29:00.989526 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:29:00.998399 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:00.991602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.98.62:43519)
0620 12:29:00.991740 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:00.991743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:00.991937 (+ 194us) server_negotiation.cc:407] Connection header received
0620 12:29:00.992087 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:00.992089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:00.992126 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:00.992193 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:00.992894 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.993386 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.994194 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:00.994404 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:00.994491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:00.995854 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:00.995856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:00.995858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:00.995879 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:00.997053 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.997416 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.997418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.997420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.997488 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:00.997709 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:00.997712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:00.997713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:00.997917 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:00.998017 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:00.998203 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 12:29:00.998291 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 12:29:01.001217 19623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:29:01.004410 19623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:29:01.019623 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.013290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34803 (local address 127.0.0.1:56434)
0620 12:29:01.013760 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.013774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:01.013927 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:01.015057 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:01.015060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:01.015077 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:01.015233 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.015241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.016790 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.016795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.017653 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.017662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.018797 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.018801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.019136 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.019202 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 12:29:01.019308 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:29:01.019464 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":261,"spinlock_wait_cycles":768,"thread_start_us":102,"threads_started":1}
I20260620 12:29:01.019680 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.013982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.0.98.3:35601)
0620 12:29:01.014383 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.014396 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.014418 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:29:01.014493 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.014497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.014708 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.014881 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:01.015172 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.016137 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.017879 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.018729 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.018827 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.018983 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.019438 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.019533 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":132,"threads_started":1}
I20260620 12:29:01.019738 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.013120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.0.98.1:35373)
0620 12:29:01.014663 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.014702 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.014721 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:29:01.014785 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.014789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.014929 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.015090 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:01.015370 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.016532 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.018022 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.018603 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.018797 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.018958 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.019437 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.019543 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1445,"thread_start_us":102,"threads_started":1}
I20260620 12:29:01.020153 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.012976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35373 (local address 127.0.0.1:40864)
0620 12:29:01.013643 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.013694 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:01.013975 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:01.015068 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:01.015074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:01.015096 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:01.015240 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.015246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.016674 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.016677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.017908 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.017915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.018802 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.018806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.018945 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.019210 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 12:29:01.019317 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 12:29:01.020021 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":653,"thread_start_us":102,"threads_started":1}
I20260620 12:29:01.020160 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.014203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.0.98.2:34803)
0620 12:29:01.014561 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.014572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.014595 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:29:01.014689 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.014697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.014892 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.015406 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:01.015574 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.016659 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.017778 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.018631 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.018980 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.019111 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.019837 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.019943 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":178,"threads_started":1}
I20260620 12:29:01.020511 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.013766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35601 (local address 127.0.0.1:39794)
0620 12:29:01.014196 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.014209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:01.014290 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:01.014867 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:01.014873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:01.014895 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:01.015037 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.015046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.016292 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.016295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.017744 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.017754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.019045 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.019049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.019261 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.019295 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:29:01.019355 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 12:29:01.020428 (+ 1073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1150,"thread_start_us":94,"threads_started":1}
I20260620 12:29:01.022933 19882 tablet_service.cc:1467] Processing CreateTablet for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 (DEFAULT_TABLE table=test-table [id=d9bb8a0130bd4ee4a1effd75a071c0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:29:01.022936 20016 tablet_service.cc:1467] Processing CreateTablet for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 (DEFAULT_TABLE table=test-table [id=d9bb8a0130bd4ee4a1effd75a071c0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:29:01.024026 19882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:01.024044 20016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:01.024501 19748 tablet_service.cc:1467] Processing CreateTablet for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 (DEFAULT_TABLE table=test-table [id=d9bb8a0130bd4ee4a1effd75a071c0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:29:01.025401 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:01.030045 20099 tablet_bootstrap.cc:492] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Bootstrap starting.
I20260620 12:29:01.031680 20100 tablet_bootstrap.cc:492] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Bootstrap starting.
I20260620 12:29:01.031962 20101 tablet_bootstrap.cc:492] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d: Bootstrap starting.
I20260620 12:29:01.032503 20099 tablet_bootstrap.cc:654] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:01.033877 20099 log.cc:824] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:01.034078 20101 tablet_bootstrap.cc:654] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:01.034114 20100 tablet_bootstrap.cc:654] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:01.034873 20100 log.cc:824] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:01.035274 20101 log.cc:824] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:01.036087 20099 tablet_bootstrap.cc:492] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: No bootstrap required, opened a new log
I20260620 12:29:01.036283 20099 ts_tablet_manager.cc:1397] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:29:01.036818 20101 tablet_bootstrap.cc:492] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d: No bootstrap required, opened a new log
I20260620 12:29:01.036959 20100 tablet_bootstrap.cc:492] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: No bootstrap required, opened a new log
I20260620 12:29:01.037034 20101 ts_tablet_manager.cc:1397] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:29:01.037132 20100 ts_tablet_manager.cc:1397] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:29:01.039834 20101 raft_consensus.cc:348] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.040107 20101 raft_consensus.cc:374] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:01.040162 20101 raft_consensus.cc:729] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed160bc77964eaeae016d5908d7228d, State: Initialized, Role: FOLLOWER
I20260620 12:29:01.040169 20099 raft_consensus.cc:348] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.040424 20099 raft_consensus.cc:374] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:01.040472 20099 raft_consensus.cc:729] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e3a96f35304c7c9d4c000b962641d7, State: Initialized, Role: FOLLOWER
I20260620 12:29:01.040650 20101 consensus_queue.cc:260] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.040894 20099 consensus_queue.cc:260] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.040899 20100 raft_consensus.cc:348] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.041134 20100 raft_consensus.cc:374] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:01.041200 20100 raft_consensus.cc:729] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e526d0a743554aeb90b3a5c4f2c55aba, State: Initialized, Role: FOLLOWER
I20260620 12:29:01.041540 19814 heartbeater.cc:503] Master 127.0.98.62:43519 was elected leader, sending a full tablet report...
I20260620 12:29:01.041744 20101 ts_tablet_manager.cc:1428] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:29:01.041705 20100 consensus_queue.cc:260] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.041868 20099 ts_tablet_manager.cc:1428] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260620 12:29:01.042166 20082 heartbeater.cc:503] Master 127.0.98.62:43519 was elected leader, sending a full tablet report...
I20260620 12:29:01.042320 19948 heartbeater.cc:503] Master 127.0.98.62:43519 was elected leader, sending a full tablet report...
I20260620 12:29:01.042606 20100 ts_tablet_manager.cc:1428] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:29:01.161092 20107 raft_consensus.cc:482] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:29:01.161309 20107 raft_consensus.cc:504] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.162424 20107 leader_election.cc:288] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed160bc77964eaeae016d5908d7228d (127.0.98.1:35373), 00e3a96f35304c7c9d4c000b962641d7 (127.0.98.3:35601)
I20260620 12:29:01.166452 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.162750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35373 (local address 127.0.98.2:38945)
0620 12:29:01.162888 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.162904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:01.163002 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:01.163359 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:01.163363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:01.163389 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:01.163556 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.163562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.164451 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.164454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.165286 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.165296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.166006 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.166010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.166123 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.166146 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:29:01.166219 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 12:29:01.166287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"spinlock_wait_cycles":3840}
I20260620 12:29:01.166518 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.162872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38945 (local address 127.0.98.1:35373)
0620 12:29:01.163009 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.163013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.163033 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:29:01.163097 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.163100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.163152 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.163235 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:01.163714 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.164331 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.165436 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.165885 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.165967 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.166016 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.166298 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.166383 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:29:01.166585 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.163177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41987 (local address 127.0.98.3:35601)
0620 12:29:01.163296 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.163300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.163445 (+ 145us) server_negotiation.cc:407] Connection header received
0620 12:29:01.163534 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.163537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.163589 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.163669 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:01.163964 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.164462 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.165568 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.166003 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.166105 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.166162 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.166379 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.166448 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:29:01.166847 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.163017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35601 (local address 127.0.98.2:41987)
0620 12:29:01.163343 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.163363 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:01.163455 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:01.163697 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:01.163700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:01.163716 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:01.163847 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.163853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.164591 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.164594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.165440 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.165449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.166123 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.166126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.166241 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.166265 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:29:01.166341 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 12:29:01.166404 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":157,"threads_started":1}
I20260620 12:29:01.167295 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aefe3a5d4f04cc5aa67f3b9306359d8" candidate_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed160bc77964eaeae016d5908d7228d" is_pre_election: true
I20260620 12:29:01.167459 20036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aefe3a5d4f04cc5aa67f3b9306359d8" candidate_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e3a96f35304c7c9d4c000b962641d7" is_pre_election: true
I20260620 12:29:01.167631 19768 raft_consensus.cc:2454] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e526d0a743554aeb90b3a5c4f2c55aba in term 0.
I20260620 12:29:01.167761 20036 raft_consensus.cc:2454] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e526d0a743554aeb90b3a5c4f2c55aba in term 0.
I20260620 12:29:01.168125 19837 leader_election.cc:302] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed160bc77964eaeae016d5908d7228d, e526d0a743554aeb90b3a5c4f2c55aba; no voters:
I20260620 12:29:01.168351 20107 raft_consensus.cc:2781] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:29:01.168444 20107 raft_consensus.cc:482] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:29:01.168509 20107 raft_consensus.cc:3037] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:01.169539 20107 raft_consensus.cc:504] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.169866 20107 leader_election.cc:288] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [CANDIDATE]: Term 1 election: Requested vote from peers 0ed160bc77964eaeae016d5908d7228d (127.0.98.1:35373), 00e3a96f35304c7c9d4c000b962641d7 (127.0.98.3:35601)
I20260620 12:29:01.170280 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aefe3a5d4f04cc5aa67f3b9306359d8" candidate_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed160bc77964eaeae016d5908d7228d"
I20260620 12:29:01.170341 20036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aefe3a5d4f04cc5aa67f3b9306359d8" candidate_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e3a96f35304c7c9d4c000b962641d7"
I20260620 12:29:01.170418 19768 raft_consensus.cc:3037] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:01.170454 20036 raft_consensus.cc:3037] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:01.171487 19768 raft_consensus.cc:2454] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e526d0a743554aeb90b3a5c4f2c55aba in term 1.
I20260620 12:29:01.171519 20036 raft_consensus.cc:2454] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e526d0a743554aeb90b3a5c4f2c55aba in term 1.
I20260620 12:29:01.171800 19837 leader_election.cc:302] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e3a96f35304c7c9d4c000b962641d7, e526d0a743554aeb90b3a5c4f2c55aba; no voters:
I20260620 12:29:01.171978 20107 raft_consensus.cc:2781] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:01.172217 20107 raft_consensus.cc:686] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 1 LEADER]: Becoming Leader. State: Replica: e526d0a743554aeb90b3a5c4f2c55aba, State: Running, Role: LEADER
I20260620 12:29:01.172531 20107 consensus_queue.cc:237] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } }
I20260620 12:29:01.175688 19622 catalog_manager.cc:5495] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba reported cstate change: term changed from 0 to 1, leader changed from <none> to e526d0a743554aeb90b3a5c4f2c55aba (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e526d0a743554aeb90b3a5c4f2c55aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34803 } health_report { overall_health: HEALTHY } } }
I20260620 12:29:01.205042 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.201452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.0.98.2:34803)
0620 12:29:01.201592 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.201597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.201611 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:29:01.201751 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.201754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.201810 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.201927 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:01.202165 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.202752 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.203614 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.203842 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.203955 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.204256 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:01.204667 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:01.204803 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.204844 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.204897 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:29:01.210551 20036 raft_consensus.cc:1261] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 1 FOLLOWER]: Refusing update from remote peer e526d0a743554aeb90b3a5c4f2c55aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:01.210551 19768 raft_consensus.cc:1261] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 1 FOLLOWER]: Refusing update from remote peer e526d0a743554aeb90b3a5c4f2c55aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:01.211270 20107 consensus_queue.cc:1035] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed160bc77964eaeae016d5908d7228d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:29:01.211443 20109 consensus_queue.cc:1035] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e3a96f35304c7c9d4c000b962641d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:29:01.215834 20083 tablet.cc:2369] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:29:01.218971 20114 mvcc.cc:205] Tried to move back new op lower bound from 7298902184791568384 to 7298902184645058560. Current Snapshot: MvccSnapshot[applied={T|T < 7298902184791568384}]
I20260620 12:29:01.221427 20113 mvcc.cc:205] Tried to move back new op lower bound from 7298902184791568384 to 7298902184645058560. Current Snapshot: MvccSnapshot[applied={T|T < 7298902184791568384}]
I20260620 12:29:01.226962 19622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:29:01.227099 19622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:29:01.229310 19622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0527fd757be441c39df7ae7106ade622: Sending DeleteTablet for 3 replicas of tablet 3aefe3a5d4f04cc5aa67f3b9306359d8
I20260620 12:29:01.230103 19882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:29:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56434
I20260620 12:29:01.230104 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:29:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40864
I20260620 12:29:01.230545 20122 tablet_replica.cc:330] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: stopping tablet replica
I20260620 12:29:01.230564 20123 tablet_replica.cc:330] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d: stopping tablet replica
I20260620 12:29:01.230895 20123 raft_consensus.cc:2229] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:29:01.230901 20122 raft_consensus.cc:2229] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:29:01.231200 20123 raft_consensus.cc:2258] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 0ed160bc77964eaeae016d5908d7228d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:29:01.230590 20016 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:29:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39794
I20260620 12:29:01.231541 20122 raft_consensus.cc:2258] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:29:01.231843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19685
I20260620 12:29:01.231850 20124 tablet_replica.cc:330] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: stopping tablet replica
I20260620 12:29:01.232185 20124 raft_consensus.cc:2229] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:29:01.232576 20124 raft_consensus.cc:2258] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:29:01.233160 20122 ts_tablet_manager.cc:1905] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:29:01.233907 20124 ts_tablet_manager.cc:1905] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:29:01.236433 20122 ts_tablet_manager.cc:1918] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:29:01.236541 20122 log.cc:1196] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/3aefe3a5d4f04cc5aa67f3b9306359d8
I20260620 12:29:01.236871 20122 ts_tablet_manager.cc:1939] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P e526d0a743554aeb90b3a5c4f2c55aba: Deleting consensus metadata
I20260620 12:29:01.237402 20124 ts_tablet_manager.cc:1918] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:29:01.237561 20124 log.cc:1196] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/3aefe3a5d4f04cc5aa67f3b9306359d8
I20260620 12:29:01.238003 20124 ts_tablet_manager.cc:1939] T 3aefe3a5d4f04cc5aa67f3b9306359d8 P 00e3a96f35304c7c9d4c000b962641d7: Deleting consensus metadata
I20260620 12:29:01.239246 19609 catalog_manager.cc:4841] TS 00e3a96f35304c7c9d4c000b962641d7 (127.0.98.3:35601): tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 (table test-table [id=d9bb8a0130bd4ee4a1effd75a071c0ea]) successfully deleted
W20260620 12:29:01.241182 19609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35373 (error 108)
I20260620 12:29:01.241369 19608 catalog_manager.cc:4841] TS e526d0a743554aeb90b3a5c4f2c55aba (127.0.98.2:34803): tablet 3aefe3a5d4f04cc5aa67f3b9306359d8 (table test-table [id=d9bb8a0130bd4ee4a1effd75a071c0ea]) successfully deleted
I20260620 12:29:01.241643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19819
I20260620 12:29:01.242733 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.242417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35373 (local address 127.0.0.1:40876)
0620 12:29:01.242538 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.242613 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 12:29:01.242905 19609 catalog_manager.cc:4584] TS 0ed160bc77964eaeae016d5908d7228d (127.0.98.1:35373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3aefe3a5d4f04cc5aa67f3b9306359d8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35373: connect: Connection refused (error 111)
I20260620 12:29:01.247931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19953
I20260620 12:29:01.253885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 19591
2026-06-20T12:29:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2312 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T12:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:01Z Disabled control of system clock
I20260620 12:29:01.283963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34269
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:29:01.394452 20129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:01.394783 20129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:01.394865 20129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:01.398828 20129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:01.398921 20129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:01.398955 20129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:01.398998 20129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:01.399041 20129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:01.403731 20129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:01.404821 20129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:01.406240 20129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:01.414605 20134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:01.414605 20137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:01.414796 20129 server_base.cc:1034] running on GCE node
W20260620 12:29:01.414949 20135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:01.415515 20129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:01.416527 20129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:01.417738 20129 hybrid_clock.cc:648] HybridClock initialized: now 1781958541417713 us; error 48 us; skew 500 ppm
I20260620 12:29:01.419600 20129 webserver.cc:460] Webserver started at http://127.0.98.62:33669/ using document root <none> and password file <none>
I20260620 12:29:01.420169 20129 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:01.420262 20129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:01.420490 20129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:01.422459 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "f7242615a2ef411ba8b8d454744d07ff"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "749a66e97e42f732747871fd52e66678"
server_key_iv: "13d8c1ac94331aca8aac2e622857094b"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.423075 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7242615a2ef411ba8b8d454744d07ff"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "749a66e97e42f732747871fd52e66678"
server_key_iv: "13d8c1ac94331aca8aac2e622857094b"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.426918 20129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:29:01.429322 20143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:01.430645 20129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:01.430848 20129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "f7242615a2ef411ba8b8d454744d07ff"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "749a66e97e42f732747871fd52e66678"
server_key_iv: "13d8c1ac94331aca8aac2e622857094b"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.430956 20129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:01.442589 20129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:01.443250 20129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:01.443437 20129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:01.450268 20129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34269
I20260620 12:29:01.450278 20195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34269 every 8 connection(s)
I20260620 12:29:01.451200 20129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:01.454200 20196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:01.459327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 20129
I20260620 12:29:01.459456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:01.459769 392 external_mini_cluster.cc:1428] Setting key 5eb04cc35468dd185e525bd778cc4c52
I20260620 12:29:01.459972 20196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff: Bootstrap starting.
I20260620 12:29:01.462491 20196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:01.463260 20196 log.cc:824] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:01.465135 20196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff: No bootstrap required, opened a new log
I20260620 12:29:01.468863 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.461099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.0.98.62:34269)
0620 12:29:01.461766 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.461780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.461813 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:29:01.462419 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.462434 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.462938 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.463265 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:01.463719 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.464775 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.465720 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.466195 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.466378 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.467220 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:01.467224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:01.467238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:01.467601 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:01.467674 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:01.467699 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:01.467817 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.467929 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.468172 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260620 12:29:01.469103 20196 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7242615a2ef411ba8b8d454744d07ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 12:29:01.469391 20196 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:01.469477 20196 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7242615a2ef411ba8b8d454744d07ff, State: Initialized, Role: FOLLOWER
I20260620 12:29:01.469983 20196 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7242615a2ef411ba8b8d454744d07ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 12:29:01.470147 20196 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:01.470248 20196 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:01.470383 20196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:01.471508 20196 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7242615a2ef411ba8b8d454744d07ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 12:29:01.471853 20196 leader_election.cc:302] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7242615a2ef411ba8b8d454744d07ff; no voters:
I20260620 12:29:01.472200 20196 leader_election.cc:288] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:01.472273 20201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:01.472537 20201 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [term 1 LEADER]: Becoming Leader. State: Replica: f7242615a2ef411ba8b8d454744d07ff, State: Running, Role: LEADER
I20260620 12:29:01.472858 20201 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7242615a2ef411ba8b8d454744d07ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 12:29:01.473434 20196 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:01.474556 20202 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7242615a2ef411ba8b8d454744d07ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7242615a2ef411ba8b8d454744d07ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } } }
I20260620 12:29:01.474561 20203 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7242615a2ef411ba8b8d454744d07ff. Latest consensus state: current_term: 1 leader_uuid: "f7242615a2ef411ba8b8d454744d07ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7242615a2ef411ba8b8d454744d07ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } } }
I20260620 12:29:01.474694 20202 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:01.474722 20203 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:01.475070 20210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:01.478091 20210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:01.483065 20210 catalog_manager.cc:1347] Generated new cluster ID: 51f2c80b73a1473b9a0ac9ff0d007778
I20260620 12:29:01.483132 20210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:01.492102 20210 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:01.492934 20210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:01.499513 20210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff: Generated new TSK 0
I20260620 12:29:01.500296 20210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:01.509882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:29:01.620425 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:01.620669 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:01.620707 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:01.624742 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:01.624794 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:01.624874 20220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:29:01.629518 20220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:01.630544 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:01.631917 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:01.641562 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:01.641578 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:01.641578 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:01.641753 20220 server_base.cc:1034] running on GCE node
I20260620 12:29:01.642223 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:01.642901 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:01.644057 20220 hybrid_clock.cc:648] HybridClock initialized: now 1781958541644029 us; error 41 us; skew 500 ppm
I20260620 12:29:01.645833 20220 webserver.cc:460] Webserver started at http://127.0.98.1:42119/ using document root <none> and password file <none>
I20260620 12:29:01.646407 20220 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:01.646476 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:01.646762 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:01.648736 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data/instance:
uuid: "047da018e2584a1585d8bf2e430868fc"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "7a52a7ac57950a16b4c14b36d2a02b8b"
server_key_iv: "e98b2f07a9bef6094ec97990cfda5e3d"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.649283 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/wal/instance:
uuid: "047da018e2584a1585d8bf2e430868fc"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "7a52a7ac57950a16b4c14b36d2a02b8b"
server_key_iv: "e98b2f07a9bef6094ec97990cfda5e3d"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.652901 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:29:01.655217 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:01.656405 20220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:01.656533 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/wal
uuid: "047da018e2584a1585d8bf2e430868fc"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "7a52a7ac57950a16b4c14b36d2a02b8b"
server_key_iv: "e98b2f07a9bef6094ec97990cfda5e3d"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.656637 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:01.679486 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:01.680152 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:01.680344 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:01.680913 20220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:29:01.681829 20220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:29:01.681882 20220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:01.681950 20220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:29:01.681983 20220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:01.690992 20220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45291
I20260620 12:29:01.691036 20347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45291 every 8 connection(s)
I20260620 12:29:01.691874 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:29:01.696319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 20220
I20260620 12:29:01.696456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:29:01.696766 392 external_mini_cluster.cc:1428] Setting key 50788d867dbf203c9eeb611cf88a01a1
I20260620 12:29:01.698726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:29:01.701270 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.693643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58641 (local address 127.0.98.62:34269)
0620 12:29:01.693787 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.693790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.694797 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 12:29:01.696197 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.696201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.696259 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.696368 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:01.697449 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.698004 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.699300 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.699541 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.699661 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.700762 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:01.700766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:01.700769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:01.700782 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:01.700810 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:01.700818 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:01.700905 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.701070 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.701133 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:29:01.702097 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.693973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34269 (local address 127.0.98.1:58641)
0620 12:29:01.694600 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.694640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:01.695973 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:01.696575 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:01.696584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:01.696684 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:01.697280 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.697303 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.698134 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.698137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.699169 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.699179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.699697 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.699701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.699866 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.700501 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:29:01.700503 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:01.700542 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:29:01.700603 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:01.700906 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:01.700918 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:01.700927 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:29:01.701065 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 12:29:01.701345 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260620 12:29:01.703383 20348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34269
I20260620 12:29:01.703646 20348 heartbeater.cc:464] Registering TS with master...
I20260620 12:29:01.704308 20348 heartbeater.cc:511] Master 127.0.98.62:34269 requested a full tablet report, sending...
I20260620 12:29:01.706090 20160 ts_manager.cc:194] Registered new tserver with Master: 047da018e2584a1585d8bf2e430868fc (127.0.98.1:45291)
I20260620 12:29:01.707971 20160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58641
W20260620 12:29:01.811959 20351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:01.812196 20351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:01.812239 20351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:01.816279 20351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:01.816337 20351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:01.816395 20351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:29:01.821013 20351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:01.821956 20351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:01.823302 20351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:01.832609 20356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:01.832602 20359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:01.832602 20357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:01.832856 20351 server_base.cc:1034] running on GCE node
I20260620 12:29:01.833318 20351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:01.833968 20351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:01.835180 20351 hybrid_clock.cc:648] HybridClock initialized: now 1781958541835127 us; error 65 us; skew 500 ppm
I20260620 12:29:01.836932 20351 webserver.cc:460] Webserver started at http://127.0.98.2:40579/ using document root <none> and password file <none>
I20260620 12:29:01.837505 20351 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:01.837596 20351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:01.837821 20351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:01.839876 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data/instance:
uuid: "5cdc25aa6a274f0981c85122499a8aba"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "71411451ce82b0203a02c95f314835c5"
server_key_iv: "f70d23080f78fb1fa2192074b193801a"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.840456 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5cdc25aa6a274f0981c85122499a8aba"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "71411451ce82b0203a02c95f314835c5"
server_key_iv: "f70d23080f78fb1fa2192074b193801a"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.843948 20351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:29:01.846267 20365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:01.847491 20351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:01.847622 20351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal
uuid: "5cdc25aa6a274f0981c85122499a8aba"
format_stamp: "Formatted at 2026-06-20 12:29:01 on dist-test-slave-bwpp"
server_key: "71411451ce82b0203a02c95f314835c5"
server_key_iv: "f70d23080f78fb1fa2192074b193801a"
server_key_version: "encryptionkey@0"
I20260620 12:29:01.847729 20351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:01.866609 20351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:01.867475 20351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:01.867654 20351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:01.868225 20351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:29:01.869138 20351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:29:01.869184 20351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:01.869251 20351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:29:01.869283 20351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:01.878095 20351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34605
I20260620 12:29:01.878165 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34605 every 8 connection(s)
I20260620 12:29:01.879043 20351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:29:01.884841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 20351
I20260620 12:29:01.884982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:29:01.885275 392 external_mini_cluster.cc:1428] Setting key 5b6b3e7be4a89a0a1028e3751b621fef
I20260620 12:29:01.887247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:29:01.888510 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.880782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36959 (local address 127.0.98.62:34269)
0620 12:29:01.880920 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:01.880923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:01.881943 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 12:29:01.883408 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:01.883412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:01.883467 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:01.883550 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:01.884408 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.884979 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.886365 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.886707 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.886814 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.887876 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:01.887883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:01.887889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:01.887907 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:01.887934 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:01.887947 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:01.888030 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:01.888162 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 12:29:01.888231 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:29:01.889540 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:01.881179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34269 (local address 127.0.98.2:36959)
0620 12:29:01.881799 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:01.881844 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:01.883175 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:01.883714 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:01.883720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:01.883807 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:01.884270 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.884282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.885119 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.885122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.886242 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:01.886252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:01.886852 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:01.886854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:01.886987 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:01.887628 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:29:01.887629 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:01.887668 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:29:01.887727 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:01.888035 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:01.888042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:01.888045 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:29:01.888133 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:29:01.888399 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260620 12:29:01.890681 20479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34269
I20260620 12:29:01.890936 20479 heartbeater.cc:464] Registering TS with master...
I20260620 12:29:01.891422 20479 heartbeater.cc:511] Master 127.0.98.62:34269 requested a full tablet report, sending...
I20260620 12:29:01.892454 20160 ts_manager.cc:194] Registered new tserver with Master: 5cdc25aa6a274f0981c85122499a8aba (127.0.98.2:34605)
I20260620 12:29:01.893340 20160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36959
W20260620 12:29:02.000305 20482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:02.000545 20482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:02.000581 20482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:02.004575 20482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:02.004626 20482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:02.004678 20482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:29:02.009306 20482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:02.010313 20482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:02.011699 20482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:02.021060 20487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:02.021068 20490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:02.021070 20488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:02.021365 20482 server_base.cc:1034] running on GCE node
I20260620 12:29:02.021773 20482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:02.022353 20482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:02.023535 20482 hybrid_clock.cc:648] HybridClock initialized: now 1781958542023510 us; error 48 us; skew 500 ppm
I20260620 12:29:02.025314 20482 webserver.cc:460] Webserver started at http://127.0.98.3:36653/ using document root <none> and password file <none>
I20260620 12:29:02.025873 20482 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:02.025964 20482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:02.026187 20482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:02.028216 20482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a5f03b1d9b7406e98e0afdcf294bacc"
format_stamp: "Formatted at 2026-06-20 12:29:02 on dist-test-slave-bwpp"
server_key: "2c89615275823cf57e27ef4b8b38a135"
server_key_iv: "66bd21d4847ae4ed11b9f092b3b88370"
server_key_version: "encryptionkey@0"
I20260620 12:29:02.028800 20482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a5f03b1d9b7406e98e0afdcf294bacc"
format_stamp: "Formatted at 2026-06-20 12:29:02 on dist-test-slave-bwpp"
server_key: "2c89615275823cf57e27ef4b8b38a135"
server_key_iv: "66bd21d4847ae4ed11b9f092b3b88370"
server_key_version: "encryptionkey@0"
I20260620 12:29:02.032280 20482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 12:29:02.034591 20496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:02.035800 20482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:02.035929 20482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal
uuid: "0a5f03b1d9b7406e98e0afdcf294bacc"
format_stamp: "Formatted at 2026-06-20 12:29:02 on dist-test-slave-bwpp"
server_key: "2c89615275823cf57e27ef4b8b38a135"
server_key_iv: "66bd21d4847ae4ed11b9f092b3b88370"
server_key_version: "encryptionkey@0"
I20260620 12:29:02.036036 20482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:02.081789 20482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:02.082527 20482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:02.082763 20482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:02.083362 20482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:29:02.084303 20482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:29:02.084347 20482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:02.084417 20482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:29:02.084450 20482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:02.093394 20482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40135
I20260620 12:29:02.093415 20609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40135 every 8 connection(s)
I20260620 12:29:02.094316 20482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:29:02.102973 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.096079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55333 (local address 127.0.98.62:34269)
0620 12:29:02.096248 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.096253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.097278 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 12:29:02.098609 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.098612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.098706 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.098783 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:02.099625 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.100146 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.101231 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.101435 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.101517 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.102510 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:02.102513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:02.102515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:02.102527 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:02.102546 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:02.102554 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:02.102610 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.102812 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.102862 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:29:02.103736 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.096510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34269 (local address 127.0.98.3:55333)
0620 12:29:02.097129 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:02.097167 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:02.098391 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:02.098938 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:02.098945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:02.099026 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:02.099502 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.099514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.100257 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.100261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.101128 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.101137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.101564 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.101566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.101700 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.102265 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:29:02.102268 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:02.102307 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:29:02.102364 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:02.102623 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:02.102629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:02.102633 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:29:02.102792 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 12:29:02.102992 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260620 12:29:02.104399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 20482
I20260620 12:29:02.104517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:29:02.104825 392 external_mini_cluster.cc:1428] Setting key 06a34b785fa816df540dc561a1128b1f
I20260620 12:29:02.104949 20610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34269
I20260620 12:29:02.105219 20610 heartbeater.cc:464] Registering TS with master...
I20260620 12:29:02.105855 20610 heartbeater.cc:511] Master 127.0.98.62:34269 requested a full tablet report, sending...
I20260620 12:29:02.107024 20159 ts_manager.cc:194] Registered new tserver with Master: 0a5f03b1d9b7406e98e0afdcf294bacc (127.0.98.3:40135)
I20260620 12:29:02.107570 20159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55333
I20260620 12:29:02.118928 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:29:02.125000 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.120907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.98.62:34269)
0620 12:29:02.121068 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.121074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.121252 (+ 178us) server_negotiation.cc:407] Connection header received
0620 12:29:02.121406 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.121409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.121466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.121559 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:02.121858 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.122377 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.123312 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.123546 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.123659 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.124558 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:02.124561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:02.124563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:02.124574 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:02.124594 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:02.124602 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:02.124679 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.124783 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.124843 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:29:02.128093 20159 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:29:02.131357 20159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:29:02.146736 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.139664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45291 (local address 127.0.0.1:38334)
0620 12:29:02.140482 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:02.140518 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:02.140660 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:02.141455 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:02.141461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:02.141498 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:02.141712 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.141718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.143179 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.143182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.144569 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.144580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.145903 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.145908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.146102 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.146179 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 12:29:02.146329 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 12:29:02.146532 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":139,"spinlock_wait_cycles":4736,"thread_start_us":93,"threads_started":1}
I20260620 12:29:02.147090 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.140972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.0.98.2:34605)
0620 12:29:02.141696 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.141709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.141732 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:29:02.141818 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.141823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.141995 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.142139 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:02.142366 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.143482 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.144818 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.145824 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.145938 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.146126 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.146803 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.146914 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":99,"threads_started":1}
I20260620 12:29:02.146736 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.140427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34605 (local address 127.0.0.1:51468)
0620 12:29:02.141321 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:02.141335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:02.141413 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:02.142137 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:02.142140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:02.142158 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:02.142233 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.142238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.143606 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.143609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.144704 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.144711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.146188 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.146192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.146277 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.146308 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:29:02.146386 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 12:29:02.146532 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260620 12:29:02.147773 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.139829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.0.98.1:45291)
0620 12:29:02.140182 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.140193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.140821 (+ 628us) server_negotiation.cc:407] Connection header received
0620 12:29:02.140904 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.140909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.141122 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.141276 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:02.141831 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.143054 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.144711 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.145744 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.145870 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.146107 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.146409 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.146611 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":134,"threads_started":1}
I20260620 12:29:02.150882 20282 tablet_service.cc:1467] Processing CreateTablet for tablet e7c28c235d7e433088dac6a5fa81bd45 (DEFAULT_TABLE table=test-table [id=053ccef1e546412f8fe78dc1f802ef53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:29:02.150882 20413 tablet_service.cc:1467] Processing CreateTablet for tablet e7c28c235d7e433088dac6a5fa81bd45 (DEFAULT_TABLE table=test-table [id=053ccef1e546412f8fe78dc1f802ef53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:29:02.151788 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.139790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40135 (local address 127.0.0.1:47902)
0620 12:29:02.140674 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:02.140687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:02.140761 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:02.141456 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:02.141461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:02.141499 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:02.141709 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.141720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.148618 (+ 6898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.148622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.149704 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.149714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.151258 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.151262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.151389 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.151412 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:29:02.151500 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:29:02.151571 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":31,"spinlock_wait_cycles":7168,"thread_start_us":94,"threads_started":1}
I20260620 12:29:02.152104 20413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7c28c235d7e433088dac6a5fa81bd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:02.152107 20282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7c28c235d7e433088dac6a5fa81bd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:02.152194 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.140470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.0.98.3:40135)
0620 12:29:02.140983 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.140994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.141016 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:29:02.141090 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.141094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.141293 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.141458 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:02.141849 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.148460 (+ 6611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.149850 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.150779 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.150884 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.151184 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.151943 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.152039 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":148,"threads_started":1}
I20260620 12:29:02.155462 20544 tablet_service.cc:1467] Processing CreateTablet for tablet e7c28c235d7e433088dac6a5fa81bd45 (DEFAULT_TABLE table=test-table [id=053ccef1e546412f8fe78dc1f802ef53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:29:02.156497 20544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7c28c235d7e433088dac6a5fa81bd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:02.159466 20627 tablet_bootstrap.cc:492] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Bootstrap starting.
I20260620 12:29:02.161747 20627 tablet_bootstrap.cc:654] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:02.162487 20627 log.cc:824] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:02.162909 20629 tablet_bootstrap.cc:492] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Bootstrap starting.
I20260620 12:29:02.162963 20628 tablet_bootstrap.cc:492] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc: Bootstrap starting.
I20260620 12:29:02.164901 20627 tablet_bootstrap.cc:492] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: No bootstrap required, opened a new log
I20260620 12:29:02.165107 20627 ts_tablet_manager.cc:1397] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:29:02.165124 20629 tablet_bootstrap.cc:654] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:02.165246 20628 tablet_bootstrap.cc:654] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:02.165916 20628 log.cc:824] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:02.165963 20629 log.cc:824] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:02.167599 20629 tablet_bootstrap.cc:492] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: No bootstrap required, opened a new log
I20260620 12:29:02.167788 20629 ts_tablet_manager.cc:1397] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:29:02.167863 20628 tablet_bootstrap.cc:492] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc: No bootstrap required, opened a new log
I20260620 12:29:02.168047 20628 ts_tablet_manager.cc:1397] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:29:02.169409 20627 raft_consensus.cc:348] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.169678 20627 raft_consensus.cc:374] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:02.169760 20627 raft_consensus.cc:729] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdc25aa6a274f0981c85122499a8aba, State: Initialized, Role: FOLLOWER
I20260620 12:29:02.170253 20627 consensus_queue.cc:260] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.170969 20479 heartbeater.cc:503] Master 127.0.98.62:34269 was elected leader, sending a full tablet report...
I20260620 12:29:02.171293 20627 ts_tablet_manager.cc:1428] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 12:29:02.172124 20628 raft_consensus.cc:348] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.172384 20628 raft_consensus.cc:374] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:02.172199 20629 raft_consensus.cc:348] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.172458 20629 raft_consensus.cc:374] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:02.172482 20628 raft_consensus.cc:729] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047da018e2584a1585d8bf2e430868fc, State: Initialized, Role: FOLLOWER
I20260620 12:29:02.172528 20629 raft_consensus.cc:729] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5f03b1d9b7406e98e0afdcf294bacc, State: Initialized, Role: FOLLOWER
I20260620 12:29:02.172880 20628 consensus_queue.cc:260] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.172950 20629 consensus_queue.cc:260] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.173547 20348 heartbeater.cc:503] Master 127.0.98.62:34269 was elected leader, sending a full tablet report...
I20260620 12:29:02.173913 20628 ts_tablet_manager.cc:1428] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:29:02.174858 20629 ts_tablet_manager.cc:1428] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 12:29:02.175249 20610 heartbeater.cc:503] Master 127.0.98.62:34269 was elected leader, sending a full tablet report...
W20260620 12:29:02.193002 20349 tablet.cc:2369] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:29:02.213220 20635 raft_consensus.cc:482] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:29:02.213405 20635 raft_consensus.cc:504] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.214510 20635 leader_election.cc:288] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047da018e2584a1585d8bf2e430868fc (127.0.98.1:45291), 5cdc25aa6a274f0981c85122499a8aba (127.0.98.2:34605)
I20260620 12:29:02.218415 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.214884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41455 (local address 127.0.98.2:34605)
0620 12:29:02.215048 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.215053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.215078 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:29:02.215270 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.215273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.215327 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.215421 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:02.215732 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.216230 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.217390 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.217868 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.217984 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.218043 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.218225 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.218291 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:29:02.218415 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.214795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34605 (local address 127.0.98.3:41455)
0620 12:29:02.215014 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:02.215030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:02.215147 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:02.215453 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:02.215456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:02.215474 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:02.215588 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.215594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.216344 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.216349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.217264 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.217274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.217988 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.217992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.218086 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.218120 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:29:02.218209 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 12:29:02.218288 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":27}
I20260620 12:29:02.218546 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.214857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42231 (local address 127.0.98.1:45291)
0620 12:29:02.214985 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.214989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.215398 (+ 409us) server_negotiation.cc:407] Connection header received
0620 12:29:02.215537 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.215540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.215591 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.215683 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:29:02.215975 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.216457 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.217575 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.217958 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.218080 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.218136 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.218321 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.218394 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:29:02.218575 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.214958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45291 (local address 127.0.98.3:42231)
0620 12:29:02.215310 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:02.215320 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:02.215434 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:02.215740 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:02.215743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:02.215759 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:29:02.215871 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.215875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.216564 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.216570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.217453 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:02.217462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.218093 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.218097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:02.218202 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.218218 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 12:29:02.218305 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:29:02.218370 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":24,"thread_start_us":121,"threads_started":1}
I20260620 12:29:02.219192 20302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c28c235d7e433088dac6a5fa81bd45" candidate_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047da018e2584a1585d8bf2e430868fc" is_pre_election: true
I20260620 12:29:02.219197 20433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c28c235d7e433088dac6a5fa81bd45" candidate_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdc25aa6a274f0981c85122499a8aba" is_pre_election: true
I20260620 12:29:02.219516 20433 raft_consensus.cc:2454] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5f03b1d9b7406e98e0afdcf294bacc in term 0.
I20260620 12:29:02.219507 20302 raft_consensus.cc:2454] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5f03b1d9b7406e98e0afdcf294bacc in term 0.
I20260620 12:29:02.220036 20499 leader_election.cc:302] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 047da018e2584a1585d8bf2e430868fc, 0a5f03b1d9b7406e98e0afdcf294bacc, 5cdc25aa6a274f0981c85122499a8aba; no voters:
I20260620 12:29:02.220301 20635 raft_consensus.cc:2781] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:29:02.220376 20635 raft_consensus.cc:482] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:29:02.220428 20635 raft_consensus.cc:3037] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:02.221467 20635 raft_consensus.cc:504] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.221742 20635 leader_election.cc:288] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [CANDIDATE]: Term 1 election: Requested vote from peers 047da018e2584a1585d8bf2e430868fc (127.0.98.1:45291), 5cdc25aa6a274f0981c85122499a8aba (127.0.98.2:34605)
I20260620 12:29:02.222123 20302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c28c235d7e433088dac6a5fa81bd45" candidate_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047da018e2584a1585d8bf2e430868fc"
I20260620 12:29:02.222123 20433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7c28c235d7e433088dac6a5fa81bd45" candidate_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cdc25aa6a274f0981c85122499a8aba"
I20260620 12:29:02.222242 20433 raft_consensus.cc:3037] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:02.222242 20302 raft_consensus.cc:3037] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:02.223512 20302 raft_consensus.cc:2454] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5f03b1d9b7406e98e0afdcf294bacc in term 1.
I20260620 12:29:02.223512 20433 raft_consensus.cc:2454] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5f03b1d9b7406e98e0afdcf294bacc in term 1.
I20260620 12:29:02.223824 20499 leader_election.cc:302] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5f03b1d9b7406e98e0afdcf294bacc, 5cdc25aa6a274f0981c85122499a8aba; no voters:
I20260620 12:29:02.223996 20635 raft_consensus.cc:2781] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:02.224255 20635 raft_consensus.cc:686] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 1 LEADER]: Becoming Leader. State: Replica: 0a5f03b1d9b7406e98e0afdcf294bacc, State: Running, Role: LEADER
I20260620 12:29:02.224589 20635 consensus_queue.cc:237] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } }
I20260620 12:29:02.227525 20159 catalog_manager.cc:5495] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a5f03b1d9b7406e98e0afdcf294bacc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a5f03b1d9b7406e98e0afdcf294bacc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 } health_report { overall_health: UNKNOWN } } }
I20260620 12:29:02.249624 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.246397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.0.98.3:40135)
0620 12:29:02.246504 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:02.246508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:02.246525 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:29:02.246761 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:02.246764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:02.246816 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:02.246902 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:02.247109 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.247583 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.248371 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:02.248572 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:02.248648 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:02.248888 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:02.249271 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:02.249358 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:02.249413 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 12:29:02.249455 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 12:29:02.255321 20302 raft_consensus.cc:1261] T e7c28c235d7e433088dac6a5fa81bd45 P 047da018e2584a1585d8bf2e430868fc [term 1 FOLLOWER]: Refusing update from remote peer 0a5f03b1d9b7406e98e0afdcf294bacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:02.255328 20433 raft_consensus.cc:1261] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 1 FOLLOWER]: Refusing update from remote peer 0a5f03b1d9b7406e98e0afdcf294bacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:02.255991 20637 consensus_queue.cc:1035] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cdc25aa6a274f0981c85122499a8aba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:29:02.256166 20635 consensus_queue.cc:1035] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "047da018e2584a1585d8bf2e430868fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:29:02.262917 20640 mvcc.cc:205] Tried to move back new op lower bound from 7298902189070979072 to 7298902188954263552. Current Snapshot: MvccSnapshot[applied={T|T < 7298902189070979072}]
I20260620 12:29:02.263492 20641 mvcc.cc:205] Tried to move back new op lower bound from 7298902189070979072 to 7298902188954263552. Current Snapshot: MvccSnapshot[applied={T|T < 7298902189070979072}]
I20260620 12:29:02.264415 20642 mvcc.cc:205] Tried to move back new op lower bound from 7298902189070979072 to 7298902188954263552. Current Snapshot: MvccSnapshot[applied={T|T < 7298902189070979072}]
I20260620 12:29:02.270402 20160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:29:02.270540 20160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:29:02.272797 20160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7242615a2ef411ba8b8d454744d07ff: Sending DeleteTablet for 3 replicas of tablet e7c28c235d7e433088dac6a5fa81bd45
I20260620 12:29:02.273463 20282 tablet_service.cc:1514] Processing DeleteTablet for tablet e7c28c235d7e433088dac6a5fa81bd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:29:02 UTC) from {username='slave'} at 127.0.0.1:38334
I20260620 12:29:02.273554 20413 tablet_service.cc:1514] Processing DeleteTablet for tablet e7c28c235d7e433088dac6a5fa81bd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:29:02 UTC) from {username='slave'} at 127.0.0.1:51468
I20260620 12:29:02.273552 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet e7c28c235d7e433088dac6a5fa81bd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:29:02 UTC) from {username='slave'} at 127.0.0.1:47902
I20260620 12:29:02.274080 20649 tablet_replica.cc:330] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: stopping tablet replica
I20260620 12:29:02.274380 20649 raft_consensus.cc:2229] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:29:02.274716 20649 raft_consensus.cc:2258] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:29:02.274734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 20220
I20260620 12:29:02.275056 20651 tablet_replica.cc:330] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: stopping tablet replica
I20260620 12:29:02.275326 20651 raft_consensus.cc:2229] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:29:02.275656 20651 raft_consensus.cc:2258] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:29:02.276805 20649 ts_tablet_manager.cc:1905] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:29:02.277083 20651 ts_tablet_manager.cc:1905] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:29:02.279613 20649 ts_tablet_manager.cc:1918] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:29:02.279709 20649 log.cc:1196] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-1/wal/wals/e7c28c235d7e433088dac6a5fa81bd45
I20260620 12:29:02.280018 20649 ts_tablet_manager.cc:1939] T e7c28c235d7e433088dac6a5fa81bd45 P 5cdc25aa6a274f0981c85122499a8aba: Deleting consensus metadata
I20260620 12:29:02.280705 20651 ts_tablet_manager.cc:1918] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:29:02.280864 20651 log.cc:1196] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Deleting WAL directory at /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781958290914605-392-0/minicluster-data/ts-2/wal/wals/e7c28c235d7e433088dac6a5fa81bd45
I20260620 12:29:02.281220 20651 ts_tablet_manager.cc:1939] T e7c28c235d7e433088dac6a5fa81bd45 P 0a5f03b1d9b7406e98e0afdcf294bacc: Deleting consensus metadata
I20260620 12:29:02.281282 20146 catalog_manager.cc:4841] TS 5cdc25aa6a274f0981c85122499a8aba (127.0.98.2:34605): tablet e7c28c235d7e433088dac6a5fa81bd45 (table test-table [id=053ccef1e546412f8fe78dc1f802ef53]) successfully deleted
I20260620 12:29:02.282367 20145 catalog_manager.cc:4841] TS 0a5f03b1d9b7406e98e0afdcf294bacc (127.0.98.3:40135): tablet e7c28c235d7e433088dac6a5fa81bd45 (table test-table [id=053ccef1e546412f8fe78dc1f802ef53]) successfully deleted
W20260620 12:29:02.284483 20144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45291 (error 108)
I20260620 12:29:02.284664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 20351
I20260620 12:29:02.285908 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:02.285611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45291 (local address 127.0.0.1:38348)
0620 12:29:02.285750 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:02.285825 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 12:29:02.286073 20144 catalog_manager.cc:4584] TS 047da018e2584a1585d8bf2e430868fc (127.0.98.1:45291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7c28c235d7e433088dac6a5fa81bd45: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45291: connect: Connection refused (error 111)
I20260620 12:29:02.291347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 20482
I20260620 12:29:02.297010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 20129
2026-06-20T12:29:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1042 ms)
[----------] 6 tests from EncryptionPolicyTest (9972 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:02 dist-test-slave-bwpp krb5kdc[20655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:02 dist-test-slave-bwpp krb5kdc[20655](info): set up 2 sockets
Jun 20 12:29:02 dist-test-slave-bwpp krb5kdc[20655](info): commencing operation
krb5kdc: starting...
W20260620 12:29:03.354853 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:03 dist-test-slave-bwpp krb5kdc[20655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:03.477494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:03.585306 20668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:03.585569 20668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:03.585610 20668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:03.589643 20668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:03.589701 20668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:03.589721 20668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:03.589773 20668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:03.589804 20668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:03.595046 20668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:03.596117 20668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:03.597155 20668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:03.605315 20674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:03.605427 20668 server_base.cc:1034] running on GCE node
W20260620 12:29:03.605315 20673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:03.605316 20676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:03.606206 20668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:03.607366 20668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:03.608567 20668 hybrid_clock.cc:648] HybridClock initialized: now 1781958543608542 us; error 37 us; skew 500 ppm
Jun 20 12:29:03 dist-test-slave-bwpp krb5kdc[20655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:03.616292 20668 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:03.617147 20668 webserver.cc:460] Webserver started at http://127.0.0.1:41783/ using document root <none> and password file <none>
I20260620 12:29:03.617691 20668 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:03.617748 20668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:03.617910 20668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:03.619974 20668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "761cdd5667df46779fbf607030d98059"
format_stamp: "Formatted at 2026-06-20 12:29:03 on dist-test-slave-bwpp"
server_key: "d25f90f254b5a2a40ca36274fda3893f"
server_key_iv: "4b0df04c319e708f533f77f132430a7f"
server_key_version: "encryptionkey@0"
I20260620 12:29:03.620500 20668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "761cdd5667df46779fbf607030d98059"
format_stamp: "Formatted at 2026-06-20 12:29:03 on dist-test-slave-bwpp"
server_key: "d25f90f254b5a2a40ca36274fda3893f"
server_key_iv: "4b0df04c319e708f533f77f132430a7f"
server_key_version: "encryptionkey@0"
I20260620 12:29:03.624174 20668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:29:03.626456 20683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:03.627705 20668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:03.627804 20668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "761cdd5667df46779fbf607030d98059"
format_stamp: "Formatted at 2026-06-20 12:29:03 on dist-test-slave-bwpp"
server_key: "d25f90f254b5a2a40ca36274fda3893f"
server_key_iv: "4b0df04c319e708f533f77f132430a7f"
server_key_version: "encryptionkey@0"
I20260620 12:29:03.627889 20668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:03.647043 20668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:03.649886 20668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:03.650050 20668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:03.656948 20668 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40751
I20260620 12:29:03.656960 20735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40751 every 8 connection(s)
I20260620 12:29:03.657872 20668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:03.660620 20736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:03.663259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 20668
I20260620 12:29:03.663407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:03.663741 392 external_mini_cluster.cc:1428] Setting key f875bad87e9f888e2689485ed789a315
I20260620 12:29:03.666854 20736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059: Bootstrap starting.
I20260620 12:29:03.669373 20736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:03.670114 20736 log.cc:824] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:03.671926 20736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059: No bootstrap required, opened a new log
Jun 20 12:29:03 dist-test-slave-bwpp krb5kdc[20655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:29:03.675093 20736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761cdd5667df46779fbf607030d98059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260620 12:29:03.675350 20736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:03.675432 20736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761cdd5667df46779fbf607030d98059, State: Initialized, Role: FOLLOWER
I20260620 12:29:03.675899 20736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761cdd5667df46779fbf607030d98059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260620 12:29:03.676043 20736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:03.676112 20736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:03.676204 20736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:03.677045 20736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761cdd5667df46779fbf607030d98059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260620 12:29:03.677366 20736 leader_election.cc:302] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 761cdd5667df46779fbf607030d98059; no voters:
I20260620 12:29:03.677685 20736 leader_election.cc:288] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:03.677772 20741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:03.678057 20741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [term 1 LEADER]: Becoming Leader. State: Replica: 761cdd5667df46779fbf607030d98059, State: Running, Role: LEADER
I20260620 12:29:03.678519 20741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761cdd5667df46779fbf607030d98059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260620 12:29:03.678927 20736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:03.679149 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:03.665171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60551 (local address 127.0.0.1:40751)
0620 12:29:03.665574 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:03.665596 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:03.665639 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:29:03.666425 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:03.666446 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:03.666890 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:03.667237 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:03.667967 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:03.669404 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:03.670294 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:03.670835 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:03.671010 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:03.673551 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:03.673556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:03.673569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:03.673595 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:03.675923 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:03.676479 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:03.676485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:03.676489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:03.676595 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:03.676943 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:03.676948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:03.676950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:03.677375 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:03.677600 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:03.678023 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 12:29:03.678240 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260620 12:29:03.679705 20741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "761cdd5667df46779fbf607030d98059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761cdd5667df46779fbf607030d98059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260620 12:29:03.679811 20741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:03.680004 20741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761cdd5667df46779fbf607030d98059. Latest consensus state: current_term: 1 leader_uuid: "761cdd5667df46779fbf607030d98059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761cdd5667df46779fbf607030d98059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260620 12:29:03.680068 20741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:03.682385 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 12:29:03.684310 20756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:03.684418 20756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:03.684497 20750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:03.685539 20750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:03.691335 20750 catalog_manager.cc:1347] Generated new cluster ID: 3368854ae82f4c26a387ce9a1c141a56
I20260620 12:29:03.691395 20750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:03.693009 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:03.684500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50805 (local address 127.0.0.1:40751)
0620 12:29:03.684604 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:03.684608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:03.684864 (+ 256us) server_negotiation.cc:407] Connection header received
0620 12:29:03.684962 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:03.684964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:03.684999 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:03.685061 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:03.685895 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:03.686434 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:03.687527 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:03.687769 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:03.687878 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:03.689270 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:03.689274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:03.689276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:03.689307 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:03.691202 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:03.691732 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:03.691736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:03.691738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:03.691822 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:03.692161 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:03.692164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:03.692166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:03.692334 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:03.692492 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:03.692753 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 12:29:03.692844 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 12:29:03.718665 20750 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:03.719662 20750 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:03.728078 20750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 761cdd5667df46779fbf607030d98059: Generated new TSK 0
I20260620 12:29:03.728741 20750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:03.734339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 20668
W20260620 12:29:03.739744 20653 connection.cc:535] client connection to 127.0.0.1:40751 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40751): Connection reset by peer (error 104)
2026-06-20T12:29:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:03 dist-test-slave-bwpp krb5kdc[20767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:03 dist-test-slave-bwpp krb5kdc[20767](info): set up 2 sockets
Jun 20 12:29:03 dist-test-slave-bwpp krb5kdc[20767](info): commencing operation
krb5kdc: starting...
W20260620 12:29:04.783763 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:04 dist-test-slave-bwpp krb5kdc[20767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:04.903826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:05.012682 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:05.012940 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:05.013027 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:05.017009 20780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:05.017071 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:05.017091 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:05.017158 20780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:05.017198 20780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:05.022390 20780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:05.023411 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:05.024713 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:05.032985 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:05.032995 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:05.033033 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:05.033350 20780 server_base.cc:1034] running on GCE node
I20260620 12:29:05.033928 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:05.035094 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:05.036252 20780 hybrid_clock.cc:648] HybridClock initialized: now 1781958545036238 us; error 25 us; skew 500 ppm
Jun 20 12:29:05 dist-test-slave-bwpp krb5kdc[20767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:05.038900 20780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:05.039732 20780 webserver.cc:460] Webserver started at http://127.0.0.1:39119/ using document root <none> and password file <none>
I20260620 12:29:05.040279 20780 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:05.040369 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:05.040597 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:05.042557 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9"
format_stamp: "Formatted at 2026-06-20 12:29:05 on dist-test-slave-bwpp"
server_key: "1eaddd02607fda320db59db5302ce3e5"
server_key_iv: "71d357f15ebe8e3bc9a855b5ef2996e7"
server_key_version: "encryptionkey@0"
I20260620 12:29:05.043166 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9"
format_stamp: "Formatted at 2026-06-20 12:29:05 on dist-test-slave-bwpp"
server_key: "1eaddd02607fda320db59db5302ce3e5"
server_key_iv: "71d357f15ebe8e3bc9a855b5ef2996e7"
server_key_version: "encryptionkey@0"
I20260620 12:29:05.046713 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:29:05.049012 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:05.050138 20780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:05.050282 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9"
format_stamp: "Formatted at 2026-06-20 12:29:05 on dist-test-slave-bwpp"
server_key: "1eaddd02607fda320db59db5302ce3e5"
server_key_iv: "71d357f15ebe8e3bc9a855b5ef2996e7"
server_key_version: "encryptionkey@0"
I20260620 12:29:05.050391 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:05.060259 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:05.062881 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:05.063073 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:05.069840 20780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37495
I20260620 12:29:05.069908 20847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37495 every 8 connection(s)
I20260620 12:29:05.070792 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:05.073422 20848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:05.078583 20848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9: Bootstrap starting.
I20260620 12:29:05.079092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 20780
I20260620 12:29:05.079219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:05.079519 392 external_mini_cluster.cc:1428] Setting key 3487f7284a55f018279fb79f1a06c9cf
I20260620 12:29:05.081225 20848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:05.082096 20848 log.cc:824] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:05.084177 20848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9: No bootstrap required, opened a new log
I20260620 12:29:05.088697 20848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37495 } }
I20260620 12:29:05.088946 20848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:05.089004 20848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f83e82e1eb64f30ad6232ff6cf1b6b9, State: Initialized, Role: FOLLOWER
I20260620 12:29:05.089234 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:05.080799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44765 (local address 127.0.0.1:37495)
0620 12:29:05.081226 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:05.081252 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:05.081294 (+ 42us) server_negotiation.cc:407] Connection header received
0620 12:29:05.082130 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:05.082152 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:05.082565 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:05.082944 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:05.083312 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:05.084705 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:05.085585 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:05.086283 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:05.086433 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:05.087262 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:05.087270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:05.087289 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:05.087801 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:05.087866 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:05.087896 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:05.088024 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:05.088198 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 12:29:05.088324 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260620 12:29:05.089453 20848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37495 } }
I20260620 12:29:05.089632 20848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:05.089718 20848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:05.089812 20848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:05.090860 20848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37495 } }
I20260620 12:29:05.091217 20848 leader_election.cc:302] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f83e82e1eb64f30ad6232ff6cf1b6b9; no voters:
I20260620 12:29:05.091595 20848 leader_election.cc:288] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:05.091691 20853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:05.091929 20853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [term 1 LEADER]: Becoming Leader. State: Replica: 7f83e82e1eb64f30ad6232ff6cf1b6b9, State: Running, Role: LEADER
I20260620 12:29:05.092310 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:05.092285 20853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37495 } }
I20260620 12:29:05.092788 20848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:05.093267 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f83e82e1eb64f30ad6232ff6cf1b6b9. Latest consensus state: current_term: 1 leader_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37495 } } }
I20260620 12:29:05.093379 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:05.093716 20855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f83e82e1eb64f30ad6232ff6cf1b6b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37495 } } }
I20260620 12:29:05.093816 20855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:05.094900 20859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:05.098091 20859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:05.099243 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:05.094512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41683 (local address 127.0.0.1:37495)
0620 12:29:05.094627 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:05.094631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:05.094884 (+ 253us) server_negotiation.cc:407] Connection header received
0620 12:29:05.095072 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:05.095074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:05.095113 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:05.095159 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:05.095658 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:05.096274 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:05.097349 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:05.097678 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:05.097793 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:05.098711 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:05.098715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:05.098718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:05.098731 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:05.098783 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:05.098798 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:05.098893 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:05.099073 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 12:29:05.099105 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:29:05.103550 20859 catalog_manager.cc:1347] Generated new cluster ID: a15cb6aa5a994ddab060f3b6478800eb
I20260620 12:29:05.103611 20859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:05.115895 20859 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:05.116786 20859 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:05.132095 20859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f83e82e1eb64f30ad6232ff6cf1b6b9: Generated new TSK 0
I20260620 12:29:05.132768 20859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:05.136335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 20780
W20260620 12:29:05.141719 20765 connection.cc:535] client connection to 127.0.0.1:37495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37495): Connection reset by peer (error 104)
2026-06-20T12:29:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:05 dist-test-slave-bwpp krb5kdc[20879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:05 dist-test-slave-bwpp krb5kdc[20879](info): set up 2 sockets
Jun 20 12:29:05 dist-test-slave-bwpp krb5kdc[20879](info): commencing operation
krb5kdc: starting...
W20260620 12:29:06.187009 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:06 dist-test-slave-bwpp krb5kdc[20879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:06.319532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:06.437510 20892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:06.437820 20892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:06.437911 20892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:06.442319 20892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:06.442425 20892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:06.442474 20892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:06.442538 20892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:06.442577 20892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:06.448278 20892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:06.449345 20892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:06.450408 20892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:06.459174 20898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:06.459236 20900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:06.459249 20892 server_base.cc:1034] running on GCE node
W20260620 12:29:06.459175 20897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:06.460043 20892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:06.461040 20892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:06.462229 20892 hybrid_clock.cc:648] HybridClock initialized: now 1781958546462213 us; error 28 us; skew 500 ppm
Jun 20 12:29:06 dist-test-slave-bwpp krb5kdc[20879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:06.465049 20892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:06.466012 20892 webserver.cc:460] Webserver started at http://127.0.0.1:42041/ using document root <none> and password file <none>
I20260620 12:29:06.466593 20892 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:06.466739 20892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:06.466979 20892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:06.469067 20892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "aaeffef0c86f45648c456c94e6f0dbdc"
format_stamp: "Formatted at 2026-06-20 12:29:06 on dist-test-slave-bwpp"
server_key: "01a4448f0725bc6dce736f4ba7cf3cc8"
server_key_iv: "b58b4445900dea22d158fc58fd9cb4f0"
server_key_version: "encryptionkey@0"
I20260620 12:29:06.469646 20892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "aaeffef0c86f45648c456c94e6f0dbdc"
format_stamp: "Formatted at 2026-06-20 12:29:06 on dist-test-slave-bwpp"
server_key: "01a4448f0725bc6dce736f4ba7cf3cc8"
server_key_iv: "b58b4445900dea22d158fc58fd9cb4f0"
server_key_version: "encryptionkey@0"
I20260620 12:29:06.473407 20892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:29:06.475760 20907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:06.476953 20892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:29:06.477097 20892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "aaeffef0c86f45648c456c94e6f0dbdc"
format_stamp: "Formatted at 2026-06-20 12:29:06 on dist-test-slave-bwpp"
server_key: "01a4448f0725bc6dce736f4ba7cf3cc8"
server_key_iv: "b58b4445900dea22d158fc58fd9cb4f0"
server_key_version: "encryptionkey@0"
I20260620 12:29:06.477205 20892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:06.481611 20892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:06.484807 20892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:06.484992 20892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:06.491860 20892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38857
I20260620 12:29:06.491892 20959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38857 every 8 connection(s)
I20260620 12:29:06.492774 20892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:06.495426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 20892
I20260620 12:29:06.495566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:06.495563 20960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:06.495885 392 external_mini_cluster.cc:1428] Setting key 2b8e6ea52d0f9647e45945618de516e2
I20260620 12:29:06.501381 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc: Bootstrap starting.
I20260620 12:29:06.502240 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:06.497196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32807 (local address 127.0.0.1:38857)
0620 12:29:06.497600 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:06.497621 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:06.499004 (+ 1383us) server_negotiation.cc:407] Connection header received
0620 12:29:06.499771 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:06.499788 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:06.500052 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:06.500375 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:06.501115 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:06.501119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:06.501131 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:06.501141 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:06.501191 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:06.501209 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:06.501352 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:06.501420 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:29:06.501571 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20260620 12:29:06.503769 20960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:06.504629 20960 log.cc:824] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:06.505185 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:06.506585 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc: No bootstrap required, opened a new log
I20260620 12:29:06.508719 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:06.507216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51897 (local address 127.0.0.1:38857)
0620 12:29:06.507334 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:06.507338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:06.507499 (+ 161us) server_negotiation.cc:407] Connection header received
0620 12:29:06.507572 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:06.507575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:06.507606 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:06.507687 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:06.508364 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:06.508367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:06.508371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:06.508383 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:06.508412 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:06.508424 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:06.508513 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:06.508571 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 12:29:06.508606 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:29:06.509974 20960 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260620 12:29:06.510174 20960 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:06.510213 20960 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaeffef0c86f45648c456c94e6f0dbdc, State: Initialized, Role: FOLLOWER
I20260620 12:29:06.510555 20960 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260620 12:29:06.510718 20960 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:06.510797 20960 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:06.510870 20960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:06.511648 20960 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260620 12:29:06.511932 20960 leader_election.cc:302] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaeffef0c86f45648c456c94e6f0dbdc; no voters:
I20260620 12:29:06.512172 20960 leader_election.cc:288] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:06.512285 20970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:06.512523 20970 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [term 1 LEADER]: Becoming Leader. State: Replica: aaeffef0c86f45648c456c94e6f0dbdc, State: Running, Role: LEADER
I20260620 12:29:06.512847 20970 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260620 12:29:06.513334 20960 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:06.514446 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } } }
I20260620 12:29:06.514580 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:06.514990 20979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:06.514950 20972 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaeffef0c86f45648c456c94e6f0dbdc. Latest consensus state: current_term: 1 leader_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaeffef0c86f45648c456c94e6f0dbdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } } }
I20260620 12:29:06.515059 20972 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:06.518093 20979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:06.523626 20979 catalog_manager.cc:1347] Generated new cluster ID: 0f1915c5333c4bb3bb4c786a6e6fe0e5
I20260620 12:29:06.523697 20979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:06.539697 20979 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:06.539999 20979 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:06.544921 20979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaeffef0c86f45648c456c94e6f0dbdc: Generated new TSK 0
I20260620 12:29:06.545511 20979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:06.556947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 20892
2026-06-20T12:29:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:06 dist-test-slave-bwpp krb5kdc[20991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:06 dist-test-slave-bwpp krb5kdc[20991](info): set up 2 sockets
Jun 20 12:29:06 dist-test-slave-bwpp krb5kdc[20991](info): commencing operation
krb5kdc: starting...
W20260620 12:29:07.591019 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:07 dist-test-slave-bwpp krb5kdc[20991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:07.706200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:07.812572 21004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:07.812814 21004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:07.812855 21004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:07.816745 21004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:07.816793 21004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:07.816812 21004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:07.816828 21004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:07.816843 21004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:07.822170 21004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:07.823138 21004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:07.824095 21004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:07.832691 21012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:07.832672 21010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:07.832659 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:07.832877 21004 server_base.cc:1034] running on GCE node
I20260620 12:29:07.833565 21004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:07.834609 21004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:07.835822 21004 hybrid_clock.cc:648] HybridClock initialized: now 1781958547835803 us; error 34 us; skew 500 ppm
Jun 20 12:29:07 dist-test-slave-bwpp krb5kdc[20991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:07.838558 21004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:07.839448 21004 webserver.cc:460] Webserver started at http://127.0.0.1:38823/ using document root <none> and password file <none>
I20260620 12:29:07.839993 21004 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:07.840082 21004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:07.840318 21004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:07.842303 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "1af8948f3d3148f282e75f1488300067"
format_stamp: "Formatted at 2026-06-20 12:29:07 on dist-test-slave-bwpp"
server_key: "8527ad6953e5bcfe73ecdf3b574fc9d6"
server_key_iv: "8de555b70a0efbbfa2099f4e9cd8c1cc"
server_key_version: "encryptionkey@0"
I20260620 12:29:07.842900 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "1af8948f3d3148f282e75f1488300067"
format_stamp: "Formatted at 2026-06-20 12:29:07 on dist-test-slave-bwpp"
server_key: "8527ad6953e5bcfe73ecdf3b574fc9d6"
server_key_iv: "8de555b70a0efbbfa2099f4e9cd8c1cc"
server_key_version: "encryptionkey@0"
I20260620 12:29:07.846419 21004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 12:29:07.848860 21019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:07.850040 21004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:07.850178 21004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "1af8948f3d3148f282e75f1488300067"
format_stamp: "Formatted at 2026-06-20 12:29:07 on dist-test-slave-bwpp"
server_key: "8527ad6953e5bcfe73ecdf3b574fc9d6"
server_key_iv: "8de555b70a0efbbfa2099f4e9cd8c1cc"
server_key_version: "encryptionkey@0"
I20260620 12:29:07.850287 21004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:07.885836 21004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:07.889082 21004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:07.889285 21004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:07.896132 21004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38137
I20260620 12:29:07.896162 21071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38137 every 8 connection(s)
I20260620 12:29:07.897029 21004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:07.899616 21072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:07.901790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21004
I20260620 12:29:07.901923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:07.902216 392 external_mini_cluster.cc:1428] Setting key af0d874379cf96d459c6f5117d65e3fc
I20260620 12:29:07.905499 21072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067: Bootstrap starting.
I20260620 12:29:07.907614 21072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:07.908309 21072 log.cc:824] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:07.910490 21072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067: No bootstrap required, opened a new log
Jun 20 12:29:07 dist-test-slave-bwpp krb5kdc[20991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:29:07.913287 21072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8948f3d3148f282e75f1488300067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260620 12:29:07.913533 21072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:07.913621 21072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af8948f3d3148f282e75f1488300067, State: Initialized, Role: FOLLOWER
I20260620 12:29:07.914041 21072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8948f3d3148f282e75f1488300067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260620 12:29:07.914183 21072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:07.914270 21072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:07.914364 21072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:07.915247 21072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8948f3d3148f282e75f1488300067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260620 12:29:07.915599 21072 leader_election.cc:302] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af8948f3d3148f282e75f1488300067; no voters:
I20260620 12:29:07.915985 21072 leader_election.cc:288] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:07.916085 21077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:07.916311 21077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [term 1 LEADER]: Becoming Leader. State: Replica: 1af8948f3d3148f282e75f1488300067, State: Running, Role: LEADER
I20260620 12:29:07.916674 21077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8948f3d3148f282e75f1488300067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260620 12:29:07.917161 21072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:07.917217 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:07.903568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35957 (local address 127.0.0.1:38137)
0620 12:29:07.903987 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:07.904002 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:07.904035 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:29:07.904724 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:07.904742 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:07.905081 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:07.905409 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:07.906201 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:07.907241 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:07.908160 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:07.908648 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:07.908829 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:07.911549 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:07.911554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:07.911567 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:07.911592 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:07.913984 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:07.914496 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:07.914502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:07.914507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:07.914627 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:07.914942 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:07.914945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:07.914947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:07.915335 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:07.915541 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:07.915821 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 12:29:07.916299 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
I20260620 12:29:07.918238 21078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1af8948f3d3148f282e75f1488300067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8948f3d3148f282e75f1488300067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } } }
I20260620 12:29:07.918357 21078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:07.918524 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af8948f3d3148f282e75f1488300067. Latest consensus state: current_term: 1 leader_uuid: "1af8948f3d3148f282e75f1488300067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8948f3d3148f282e75f1488300067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } } }
I20260620 12:29:07.918632 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:07.918890 21086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:07.920197 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:07.921963 21086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:07.927714 21086 catalog_manager.cc:1347] Generated new cluster ID: 75da976f7d224e8fb68c3bdebe956adb
I20260620 12:29:07.927771 21086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:07.930884 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:07.922727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56905 (local address 127.0.0.1:38137)
0620 12:29:07.922827 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:07.922831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:07.922927 (+ 96us) server_negotiation.cc:407] Connection header received
0620 12:29:07.923059 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:07.923064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:07.923110 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:07.923183 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:07.923858 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:07.924605 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:07.925770 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:07.926078 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:07.926195 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:07.927632 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:07.927636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:07.927638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:07.927661 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:07.929304 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:07.929718 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:07.929733 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:07.929735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:07.929800 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:07.930068 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:07.930071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:07.930072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:07.930242 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:07.930323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:07.930621 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 12:29:07.930751 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 12:29:07.950496 21086 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:07.951745 21086 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:07.965646 21086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1af8948f3d3148f282e75f1488300067: Generated new TSK 0
I20260620 12:29:07.966373 21086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:07.975656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21004
2026-06-20T12:29:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:08 dist-test-slave-bwpp krb5kdc[21103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:08 dist-test-slave-bwpp krb5kdc[21103](info): set up 2 sockets
Jun 20 12:29:08 dist-test-slave-bwpp krb5kdc[21103](info): commencing operation
krb5kdc: starting...
W20260620 12:29:09.019001 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:09 dist-test-slave-bwpp krb5kdc[21103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:09.137035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:09.245340 21116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:09.245631 21116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:09.245707 21116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:09.249677 21116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:09.249838 21116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:09.249958 21116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:09.250001 21116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:09.250038 21116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:09.255412 21116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:09.256415 21116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:09.257432 21116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:09.265811 21122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:09.265822 21124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:09.265822 21121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:09.266062 21116 server_base.cc:1034] running on GCE node
I20260620 12:29:09.266701 21116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:09.267719 21116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:09.268908 21116 hybrid_clock.cc:648] HybridClock initialized: now 1781958549268878 us; error 44 us; skew 500 ppm
Jun 20 12:29:09 dist-test-slave-bwpp krb5kdc[21103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:09.274580 21116 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:09.275449 21116 webserver.cc:460] Webserver started at http://127.0.0.1:39501/ using document root <none> and password file <none>
I20260620 12:29:09.275998 21116 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:09.276087 21116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:09.276314 21116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:09.278258 21116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "2ff4a6773cd346aa823431e73f292a07"
format_stamp: "Formatted at 2026-06-20 12:29:09 on dist-test-slave-bwpp"
server_key: "5e76c82a8d15cad8db2e3d39ff86749b"
server_key_iv: "47294b73ad59a37e42834a903c410443"
server_key_version: "encryptionkey@0"
I20260620 12:29:09.278872 21116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ff4a6773cd346aa823431e73f292a07"
format_stamp: "Formatted at 2026-06-20 12:29:09 on dist-test-slave-bwpp"
server_key: "5e76c82a8d15cad8db2e3d39ff86749b"
server_key_iv: "47294b73ad59a37e42834a903c410443"
server_key_version: "encryptionkey@0"
I20260620 12:29:09.282356 21116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 12:29:09.284778 21131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:09.286130 21116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:09.286249 21116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "2ff4a6773cd346aa823431e73f292a07"
format_stamp: "Formatted at 2026-06-20 12:29:09 on dist-test-slave-bwpp"
server_key: "5e76c82a8d15cad8db2e3d39ff86749b"
server_key_iv: "47294b73ad59a37e42834a903c410443"
server_key_version: "encryptionkey@0"
I20260620 12:29:09.286346 21116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:09.300768 21116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:09.303969 21116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:09.304163 21116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:09.310832 21116 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35027
I20260620 12:29:09.310847 21183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35027 every 8 connection(s)
I20260620 12:29:09.311750 21116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:09.312656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21116
I20260620 12:29:09.312814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:09.313138 392 external_mini_cluster.cc:1428] Setting key 745ce200a73fe0f2f1041713d5ac5eb1
I20260620 12:29:09.314925 21184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:09.321797 21184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07: Bootstrap starting.
I20260620 12:29:09.323315 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:09.314555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52275 (local address 127.0.0.1:35027)
0620 12:29:09.315059 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:09.315077 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:09.315120 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:29:09.315952 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:09.315980 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:09.316390 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:09.316743 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:09.317149 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:09.318624 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:09.319525 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:09.320207 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:09.320381 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:09.321253 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:09.321259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:09.321275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:09.321772 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:09.321840 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:09.321873 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:09.321984 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:09.322185 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 12:29:09.322384 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":147,"threads_started":1}
I20260620 12:29:09.324467 21184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:09.325289 21184 log.cc:824] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:09.326197 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:09.327396 21184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07: No bootstrap required, opened a new log
I20260620 12:29:09.331761 21184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff4a6773cd346aa823431e73f292a07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 12:29:09.332046 21184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:09.332159 21184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff4a6773cd346aa823431e73f292a07, State: Initialized, Role: FOLLOWER
I20260620 12:29:09.332305 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:09.328191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46001 (local address 127.0.0.1:35027)
0620 12:29:09.328318 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:09.328321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:09.328475 (+ 154us) server_negotiation.cc:407] Connection header received
0620 12:29:09.328579 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:09.328581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:09.328620 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:09.328683 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:09.328960 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:09.329484 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:09.330645 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:09.330923 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:09.331003 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:09.331772 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:09.331775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:09.331777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:09.331787 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:09.331810 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:09.331819 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:09.331877 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:09.332004 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 12:29:09.332075 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:29:09.332646 21184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff4a6773cd346aa823431e73f292a07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 12:29:09.332772 21184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:09.332854 21184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:09.332948 21184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:09.333894 21184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff4a6773cd346aa823431e73f292a07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 12:29:09.334232 21184 leader_election.cc:302] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff4a6773cd346aa823431e73f292a07; no voters:
I20260620 12:29:09.334563 21184 leader_election.cc:288] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:09.334725 21194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:09.335049 21194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [term 1 LEADER]: Becoming Leader. State: Replica: 2ff4a6773cd346aa823431e73f292a07, State: Running, Role: LEADER
I20260620 12:29:09.335404 21194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff4a6773cd346aa823431e73f292a07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } }
I20260620 12:29:09.335770 21184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:09.337224 21195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ff4a6773cd346aa823431e73f292a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff4a6773cd346aa823431e73f292a07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } } }
I20260620 12:29:09.337364 21195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:09.337788 21196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff4a6773cd346aa823431e73f292a07. Latest consensus state: current_term: 1 leader_uuid: "2ff4a6773cd346aa823431e73f292a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff4a6773cd346aa823431e73f292a07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35027 } } }
I20260620 12:29:09.337903 21196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07 [sys.catalog]: This master's current role is: LEADER
W20260620 12:29:09.340804 21209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:09.340880 21209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:09.340997 21205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:09.341827 21205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:09.347277 21205 catalog_manager.cc:1347] Generated new cluster ID: 98a0c185afcf43c2805ee7924220d9df
I20260620 12:29:09.347340 21205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:09.366197 21205 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:09.367169 21205 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:09.376715 21205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ff4a6773cd346aa823431e73f292a07: Generated new TSK 0
I20260620 12:29:09.377403 21205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:09.382097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21116
2026-06-20T12:29:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:09 dist-test-slave-bwpp krb5kdc[21215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:09 dist-test-slave-bwpp krb5kdc[21215](info): set up 2 sockets
Jun 20 12:29:09 dist-test-slave-bwpp krb5kdc[21215](info): commencing operation
krb5kdc: starting...
W20260620 12:29:10.414918 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:10 dist-test-slave-bwpp krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:10.531589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:10.640549 21228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:10.640841 21228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:10.640946 21228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:10.644853 21228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:10.644949 21228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:10.644999 21228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:10.645063 21228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:10.645123 21228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:10.650233 21228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:10.651260 21228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:10.652247 21228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:10.660406 21234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:10.660408 21236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:10.660420 21233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:10.660673 21228 server_base.cc:1034] running on GCE node
I20260620 12:29:10.661310 21228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:10.662323 21228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:10.663480 21228 hybrid_clock.cc:648] HybridClock initialized: now 1781958550663453 us; error 38 us; skew 500 ppm
Jun 20 12:29:10 dist-test-slave-bwpp krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:10.666301 21228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:10.667276 21228 webserver.cc:460] Webserver started at http://127.0.0.1:40203/ using document root <none> and password file <none>
I20260620 12:29:10.667804 21228 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:10.667892 21228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:10.668115 21228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:10.670094 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "23380c7742c04cef833708ed3bdcae11"
format_stamp: "Formatted at 2026-06-20 12:29:10 on dist-test-slave-bwpp"
server_key: "2a2069895cac75ab887971b81fa26659"
server_key_iv: "c4b15f13699a3e5483554e467fa61c08"
server_key_version: "encryptionkey@0"
I20260620 12:29:10.670693 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "23380c7742c04cef833708ed3bdcae11"
format_stamp: "Formatted at 2026-06-20 12:29:10 on dist-test-slave-bwpp"
server_key: "2a2069895cac75ab887971b81fa26659"
server_key_iv: "c4b15f13699a3e5483554e467fa61c08"
server_key_version: "encryptionkey@0"
I20260620 12:29:10.674182 21228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:29:10.676741 21243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:10.677949 21228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:29:10.678081 21228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "23380c7742c04cef833708ed3bdcae11"
format_stamp: "Formatted at 2026-06-20 12:29:10 on dist-test-slave-bwpp"
server_key: "2a2069895cac75ab887971b81fa26659"
server_key_iv: "c4b15f13699a3e5483554e467fa61c08"
server_key_version: "encryptionkey@0"
I20260620 12:29:10.678181 21228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:10.682739 21228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:10.685473 21228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:10.685659 21228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:10.692937 21228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40479
I20260620 12:29:10.692958 21295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40479 every 8 connection(s)
I20260620 12:29:10.693854 21228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:10.696492 21296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:10.697057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21228
I20260620 12:29:10.697196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:10.697542 392 external_mini_cluster.cc:1428] Setting key 000a43a376865f81a2535b9235884c73
I20260620 12:29:10.702628 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:10.698963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40489 (local address 127.0.0.1:40479)
0620 12:29:10.699399 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:10.699412 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:10.699446 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:29:10.700099 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:10.700116 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:10.700393 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:10.700738 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:10.701524 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:10.701528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:10.701540 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:10.701550 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:10.701602 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:10.701621 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:10.701777 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:10.701847 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 12:29:10.701999 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":124,"threads_started":1}
I20260620 12:29:10.703650 21296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11: Bootstrap starting.
I20260620 12:29:10.705412 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:10.706028 21296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:10.706807 21296 log.cc:824] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:10.709038 21296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11: No bootstrap required, opened a new log
I20260620 12:29:10.709182 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:10.707535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55753 (local address 127.0.0.1:40479)
0620 12:29:10.707637 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:10.707641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:10.707877 (+ 236us) server_negotiation.cc:407] Connection header received
0620 12:29:10.707989 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:10.707991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:10.708020 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:10.708084 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:10.708751 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:10.708754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:10.708756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:10.708766 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:10.708789 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:10.708799 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:10.708863 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:10.708981 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 12:29:10.709010 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 12:29:10.712545 21296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23380c7742c04cef833708ed3bdcae11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260620 12:29:10.712751 21296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:10.712838 21296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23380c7742c04cef833708ed3bdcae11, State: Initialized, Role: FOLLOWER
I20260620 12:29:10.713276 21296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23380c7742c04cef833708ed3bdcae11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260620 12:29:10.713424 21296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:10.713493 21296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:10.713590 21296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:10.714429 21296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23380c7742c04cef833708ed3bdcae11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260620 12:29:10.714816 21296 leader_election.cc:302] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23380c7742c04cef833708ed3bdcae11; no voters:
I20260620 12:29:10.715130 21296 leader_election.cc:288] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:10.715235 21306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:10.715517 21306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [term 1 LEADER]: Becoming Leader. State: Replica: 23380c7742c04cef833708ed3bdcae11, State: Running, Role: LEADER
I20260620 12:29:10.715865 21306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23380c7742c04cef833708ed3bdcae11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } }
I20260620 12:29:10.716367 21296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:10.717494 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23380c7742c04cef833708ed3bdcae11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23380c7742c04cef833708ed3bdcae11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } } }
I20260620 12:29:10.717623 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:10.717984 21308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23380c7742c04cef833708ed3bdcae11. Latest consensus state: current_term: 1 leader_uuid: "23380c7742c04cef833708ed3bdcae11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23380c7742c04cef833708ed3bdcae11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40479 } } }
I20260620 12:29:10.718133 21308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:10.718056 21315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:10.721213 21315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:10.726634 21315 catalog_manager.cc:1347] Generated new cluster ID: a07b39c22fd04b9bbe1239b07cde665e
I20260620 12:29:10.726737 21315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:10.734573 21315 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:10.734885 21315 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:10.740025 21315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23380c7742c04cef833708ed3bdcae11: Generated new TSK 0
I20260620 12:29:10.740613 21315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:10.747670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21228
2026-06-20T12:29:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:10 dist-test-slave-bwpp krb5kdc[21327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:10 dist-test-slave-bwpp krb5kdc[21327](info): set up 2 sockets
Jun 20 12:29:10 dist-test-slave-bwpp krb5kdc[21327](info): commencing operation
krb5kdc: starting...
W20260620 12:29:11.786968 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:11 dist-test-slave-bwpp krb5kdc[21327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:11.905727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:12.021533 21340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:12.021821 21340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:12.021900 21340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:12.025831 21340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:12.025923 21340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:12.025962 21340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:12.026018 21340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:12.026054 21340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:12.031292 21340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:12.032334 21340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:12.033336 21340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:12.041649 21348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:12.041620 21346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:12.041682 21340 server_base.cc:1034] running on GCE node
W20260620 12:29:12.041627 21345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:12.042405 21340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:12.043601 21340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:12.044785 21340 hybrid_clock.cc:648] HybridClock initialized: now 1781958552044757 us; error 43 us; skew 500 ppm
Jun 20 12:29:12 dist-test-slave-bwpp krb5kdc[21327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:12.047505 21340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:12.048331 21340 webserver.cc:460] Webserver started at http://127.0.0.1:43861/ using document root <none> and password file <none>
I20260620 12:29:12.048910 21340 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:12.049002 21340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:12.049234 21340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:12.051364 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "e62523cf337746778905dca66993a146"
format_stamp: "Formatted at 2026-06-20 12:29:12 on dist-test-slave-bwpp"
server_key: "05074f401f21197ad8b8e9a70582fc7b"
server_key_iv: "a38790ee848cc11c36bf0cea11b9fc81"
server_key_version: "encryptionkey@0"
I20260620 12:29:12.051911 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "e62523cf337746778905dca66993a146"
format_stamp: "Formatted at 2026-06-20 12:29:12 on dist-test-slave-bwpp"
server_key: "05074f401f21197ad8b8e9a70582fc7b"
server_key_iv: "a38790ee848cc11c36bf0cea11b9fc81"
server_key_version: "encryptionkey@0"
I20260620 12:29:12.055545 21340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:29:12.057828 21355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:12.059017 21340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:12.059151 21340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "e62523cf337746778905dca66993a146"
format_stamp: "Formatted at 2026-06-20 12:29:12 on dist-test-slave-bwpp"
server_key: "05074f401f21197ad8b8e9a70582fc7b"
server_key_iv: "a38790ee848cc11c36bf0cea11b9fc81"
server_key_version: "encryptionkey@0"
I20260620 12:29:12.059260 21340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:12.074822 21340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:12.077701 21340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:12.077904 21340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:12.085237 21340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33273
I20260620 12:29:12.085263 21407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33273 every 8 connection(s)
I20260620 12:29:12.086205 21340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:12.089003 21408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:12.092043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21340
I20260620 12:29:12.092204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:12.092474 392 external_mini_cluster.cc:1428] Setting key 2f2d656a350b3350f292c38d2fa8d651
I20260620 12:29:12.095253 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146: Bootstrap starting.
I20260620 12:29:12.097959 21408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:12.098857 21408 log.cc:824] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:12.100922 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146: No bootstrap required, opened a new log
Jun 20 12:29:12 dist-test-slave-bwpp krb5kdc[21327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:29:12.104669 21408 raft_consensus.cc:348] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62523cf337746778905dca66993a146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260620 12:29:12.104923 21408 raft_consensus.cc:374] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:12.104988 21408 raft_consensus.cc:729] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62523cf337746778905dca66993a146, State: Initialized, Role: FOLLOWER
I20260620 12:29:12.105479 21408 consensus_queue.cc:260] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62523cf337746778905dca66993a146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260620 12:29:12.105644 21408 raft_consensus.cc:388] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:12.105723 21408 raft_consensus.cc:482] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:12.105845 21408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:12.106976 21408 raft_consensus.cc:504] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62523cf337746778905dca66993a146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260620 12:29:12.107343 21408 leader_election.cc:302] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e62523cf337746778905dca66993a146; no voters:
I20260620 12:29:12.107715 21408 leader_election.cc:288] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:12.107791 21413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:12.108000 21413 raft_consensus.cc:686] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [term 1 LEADER]: Becoming Leader. State: Replica: e62523cf337746778905dca66993a146, State: Running, Role: LEADER
I20260620 12:29:12.108299 21413 consensus_queue.cc:237] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62523cf337746778905dca66993a146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260620 12:29:12.108577 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:12.093923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:37307 (local address 127.0.0.1:33273)
0620 12:29:12.094352 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:12.094371 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:12.094414 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:29:12.095244 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:12.095280 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:12.095713 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:12.096071 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:12.096735 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:12.098211 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:12.099186 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:12.099768 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:12.099913 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:12.102803 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:12.102808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:12.102821 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:12.102847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:12.105199 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:12.105830 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:12.105836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:12.105843 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:12.106000 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:12.106358 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:12.106360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:12.106361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:12.106817 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:12.106991 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:12.107311 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 12:29:12.107561 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260620 12:29:12.108982 21408 sys_catalog.cc:564] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:12.109858 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e62523cf337746778905dca66993a146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62523cf337746778905dca66993a146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } } }
I20260620 12:29:12.110029 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:12.110287 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e62523cf337746778905dca66993a146. Latest consensus state: current_term: 1 leader_uuid: "e62523cf337746778905dca66993a146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62523cf337746778905dca66993a146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } } }
I20260620 12:29:12.110373 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:12.111835 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:12.111728 21422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:12.114729 21422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:12.120909 21422 catalog_manager.cc:1347] Generated new cluster ID: cb412c52947545058103d68afc31c2a9
I20260620 12:29:12.120973 21422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:12.122184 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:12.113752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:51019 (local address 127.0.0.1:33273)
0620 12:29:12.113987 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:12.113994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:12.114039 (+ 45us) server_negotiation.cc:407] Connection header received
0620 12:29:12.114191 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:12.114195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:12.114238 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:12.114322 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:12.115076 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:12.115591 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:12.116382 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:12.116666 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:12.116767 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:12.118427 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:12.118431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:12.118434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:12.118459 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:12.120210 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:12.120813 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:12.120816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:12.120818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:12.120903 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:12.121318 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:12.121321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:12.121323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:12.121499 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:12.121648 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:12.121915 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 12:29:12.122007 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260620 12:29:12.135828 21422 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:12.136735 21422 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:12.141760 21422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e62523cf337746778905dca66993a146: Generated new TSK 0
I20260620 12:29:12.142535 21422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:12.149535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21340
W20260620 12:29:12.156015 21325 connection.cc:535] client connection to 127.0.0.1:33273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33273): Connection reset by peer (error 104)
2026-06-20T12:29:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:12 dist-test-slave-bwpp krb5kdc[21439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:12 dist-test-slave-bwpp krb5kdc[21439](info): set up 2 sockets
Jun 20 12:29:12 dist-test-slave-bwpp krb5kdc[21439](info): commencing operation
krb5kdc: starting...
W20260620 12:29:13.182911 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:13 dist-test-slave-bwpp krb5kdc[21439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:13.301928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:13.412034 21452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:13.412287 21452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:13.412328 21452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:13.416338 21452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:13.416395 21452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:13.416414 21452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:13.416436 21452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:13.416486 21452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:13.421721 21452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:13.422801 21452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:13.423813 21452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:13.432166 21460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:13.432204 21457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:13.432478 21452 server_base.cc:1034] running on GCE node
W20260620 12:29:13.432605 21458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:13.433192 21452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:13.434202 21452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:13.435353 21452 hybrid_clock.cc:648] HybridClock initialized: now 1781958553435323 us; error 43 us; skew 500 ppm
Jun 20 12:29:13 dist-test-slave-bwpp krb5kdc[21439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:13.438090 21452 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:13.439028 21452 webserver.cc:460] Webserver started at http://127.0.0.1:39867/ using document root <none> and password file <none>
I20260620 12:29:13.439571 21452 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:13.439659 21452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:13.439882 21452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:13.441847 21452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "22f466d10eab414c9666b74aec695119"
format_stamp: "Formatted at 2026-06-20 12:29:13 on dist-test-slave-bwpp"
server_key: "c91002830f9b07fdd35701afbf2245ad"
server_key_iv: "440f83d2f996826189acd14077255531"
server_key_version: "encryptionkey@0"
I20260620 12:29:13.442407 21452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "22f466d10eab414c9666b74aec695119"
format_stamp: "Formatted at 2026-06-20 12:29:13 on dist-test-slave-bwpp"
server_key: "c91002830f9b07fdd35701afbf2245ad"
server_key_iv: "440f83d2f996826189acd14077255531"
server_key_version: "encryptionkey@0"
I20260620 12:29:13.445925 21452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:29:13.448310 21467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:13.449469 21452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:29:13.449601 21452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "22f466d10eab414c9666b74aec695119"
format_stamp: "Formatted at 2026-06-20 12:29:13 on dist-test-slave-bwpp"
server_key: "c91002830f9b07fdd35701afbf2245ad"
server_key_iv: "440f83d2f996826189acd14077255531"
server_key_version: "encryptionkey@0"
I20260620 12:29:13.449710 21452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:13.460156 21452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:13.463230 21452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:13.463424 21452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:13.470168 21452 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34229
I20260620 12:29:13.470194 21519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34229 every 8 connection(s)
I20260620 12:29:13.471109 21452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:13.473668 21520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:13.478102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21452
I20260620 12:29:13.478245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:13.478552 392 external_mini_cluster.cc:1428] Setting key e33a28a925b12dd7f97d2b8595086f87
I20260620 12:29:13.479230 21520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119: Bootstrap starting.
I20260620 12:29:13.481572 21520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:13.482296 21520 log.cc:824] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:13.484135 21520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119: No bootstrap required, opened a new log
I20260620 12:29:13.487490 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:13.479959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:50491 (local address 127.0.0.1:34229)
0620 12:29:13.480377 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:13.480391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:13.480425 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:29:13.481044 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:13.481061 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:13.481397 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:13.481724 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:13.482146 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:13.483365 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:13.484258 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:13.484727 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:13.484906 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:13.485824 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:13.485829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:13.485841 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:13.486220 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:13.486273 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:13.486295 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:13.486435 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:13.486632 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 12:29:13.486790 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
I20260620 12:29:13.487751 21520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f466d10eab414c9666b74aec695119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260620 12:29:13.488022 21520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:13.488090 21520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f466d10eab414c9666b74aec695119, State: Initialized, Role: FOLLOWER
I20260620 12:29:13.488510 21520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f466d10eab414c9666b74aec695119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260620 12:29:13.488658 21520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:13.488744 21520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:13.488880 21520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:13.490024 21520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f466d10eab414c9666b74aec695119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260620 12:29:13.490398 21520 leader_election.cc:302] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f466d10eab414c9666b74aec695119; no voters:
I20260620 12:29:13.490542 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:13.490777 21520 leader_election.cc:288] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:13.490830 21525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:13.491026 21525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [term 1 LEADER]: Becoming Leader. State: Replica: 22f466d10eab414c9666b74aec695119, State: Running, Role: LEADER
I20260620 12:29:13.491324 21525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f466d10eab414c9666b74aec695119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260620 12:29:13.492019 21520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:13.493578 21526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f466d10eab414c9666b74aec695119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f466d10eab414c9666b74aec695119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } } }
I20260620 12:29:13.493710 21526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:13.493570 21527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f466d10eab414c9666b74aec695119. Latest consensus state: current_term: 1 leader_uuid: "22f466d10eab414c9666b74aec695119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f466d10eab414c9666b74aec695119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } } }
I20260620 12:29:13.494098 21544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:13.494513 21527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:13.497481 21544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:13.498299 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:13.492715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:58803 (local address 127.0.0.1:34229)
0620 12:29:13.493757 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:13.493763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:13.493778 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:29:13.493815 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:13.493818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:13.493867 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:13.493961 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:13.494306 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:13.495104 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:13.496177 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:13.496520 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:13.496623 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:13.497691 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:13.497695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:13.497697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:13.497709 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:13.497737 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:13.497749 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:13.497825 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:13.498133 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 12:29:13.498166 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968}
I20260620 12:29:13.502974 21544 catalog_manager.cc:1347] Generated new cluster ID: 420d58711b1b4b98b70271cac88e1a87
I20260620 12:29:13.503038 21544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:13.515869 21544 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:13.516868 21544 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:13.530076 21544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f466d10eab414c9666b74aec695119: Generated new TSK 0
I20260620 12:29:13.530933 21544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:13.534777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21452
W20260620 12:29:13.539544 21437 connection.cc:535] client connection to 127.0.0.1:34229 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34229): Connection reset by peer (error 104)
2026-06-20T12:29:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:13 dist-test-slave-bwpp krb5kdc[21551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:13 dist-test-slave-bwpp krb5kdc[21551](info): set up 2 sockets
Jun 20 12:29:13 dist-test-slave-bwpp krb5kdc[21551](info): commencing operation
krb5kdc: starting...
W20260620 12:29:14.570886 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:14 dist-test-slave-bwpp krb5kdc[21551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:14.689828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:14.800341 21564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:14.800671 21564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:14.800757 21564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:14.804905 21564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:14.805001 21564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:14.805044 21564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:14.805105 21564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:14.805148 21564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:14.810781 21564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:14.811897 21564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:14.812965 21564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:14.822217 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:14.822291 21572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:14.822217 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:14.822783 21564 server_base.cc:1034] running on GCE node
I20260620 12:29:14.823318 21564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:14.824424 21564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:14.825615 21564 hybrid_clock.cc:648] HybridClock initialized: now 1781958554825602 us; error 26 us; skew 500 ppm
Jun 20 12:29:14 dist-test-slave-bwpp krb5kdc[21551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:14.832058 21564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:14.833034 21564 webserver.cc:460] Webserver started at http://127.0.0.1:40115/ using document root <none> and password file <none>
I20260620 12:29:14.833606 21564 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:14.833695 21564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:14.833938 21564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:14.836048 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "38f5f0210a684ca2b9f5e4e09429d505"
format_stamp: "Formatted at 2026-06-20 12:29:14 on dist-test-slave-bwpp"
server_key: "a63433a124c66493b9ed3b04da2fb69a"
server_key_iv: "50236c83f4c1efec7453773685521ed8"
server_key_version: "encryptionkey@0"
I20260620 12:29:14.836615 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "38f5f0210a684ca2b9f5e4e09429d505"
format_stamp: "Formatted at 2026-06-20 12:29:14 on dist-test-slave-bwpp"
server_key: "a63433a124c66493b9ed3b04da2fb69a"
server_key_iv: "50236c83f4c1efec7453773685521ed8"
server_key_version: "encryptionkey@0"
I20260620 12:29:14.840380 21564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 12:29:14.842804 21579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:14.844058 21564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:14.844201 21564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "38f5f0210a684ca2b9f5e4e09429d505"
format_stamp: "Formatted at 2026-06-20 12:29:14 on dist-test-slave-bwpp"
server_key: "a63433a124c66493b9ed3b04da2fb69a"
server_key_iv: "50236c83f4c1efec7453773685521ed8"
server_key_version: "encryptionkey@0"
I20260620 12:29:14.844312 21564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:14.848842 21564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:14.852115 21564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:14.852319 21564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:14.859320 21564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44979
I20260620 12:29:14.859346 21631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44979 every 8 connection(s)
I20260620 12:29:14.860236 21564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:14.863132 21632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:14.866050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21564
I20260620 12:29:14.866202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:14.866568 392 external_mini_cluster.cc:1428] Setting key 8c1e198b0eec4eb993c7112ef0059cb0
I20260620 12:29:14.869680 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505: Bootstrap starting.
I20260620 12:29:14.871848 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:14.868072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:36729 (local address 127.0.0.1:44979)
0620 12:29:14.868548 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:14.868562 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:14.868597 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:29:14.869168 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:14.869183 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:14.869490 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:14.869822 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:14.870625 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:14.870630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:14.870641 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:14.870694 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:14.870761 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:14.870783 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:14.870925 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:14.870995 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 12:29:14.871161 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":139,"threads_started":1}
I20260620 12:29:14.872321 21632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:14.873251 21632 log.cc:824] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:14.874768 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:14.875371 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505: No bootstrap required, opened a new log
I20260620 12:29:14.878532 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:14.876883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:45499 (local address 127.0.0.1:44979)
0620 12:29:14.876994 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:14.876999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:14.877158 (+ 159us) server_negotiation.cc:407] Connection header received
0620 12:29:14.877295 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:14.877297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:14.877325 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:14.877391 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:14.878115 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:14.878118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:14.878120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:14.878131 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:14.878154 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:14.878164 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:14.878232 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:14.878358 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 12:29:14.878391 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:29:14.879784 21632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f5f0210a684ca2b9f5e4e09429d505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260620 12:29:14.880023 21632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:14.880088 21632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f5f0210a684ca2b9f5e4e09429d505, State: Initialized, Role: FOLLOWER
I20260620 12:29:14.880577 21632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f5f0210a684ca2b9f5e4e09429d505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260620 12:29:14.880730 21632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:14.880797 21632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:14.880893 21632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:14.881810 21632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f5f0210a684ca2b9f5e4e09429d505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260620 12:29:14.882159 21632 leader_election.cc:302] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f5f0210a684ca2b9f5e4e09429d505; no voters:
I20260620 12:29:14.882525 21632 leader_election.cc:288] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:14.882594 21642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:14.882905 21642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [term 1 LEADER]: Becoming Leader. State: Replica: 38f5f0210a684ca2b9f5e4e09429d505, State: Running, Role: LEADER
I20260620 12:29:14.883291 21642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f5f0210a684ca2b9f5e4e09429d505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260620 12:29:14.883734 21632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:14.884990 21644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f5f0210a684ca2b9f5e4e09429d505. Latest consensus state: current_term: 1 leader_uuid: "38f5f0210a684ca2b9f5e4e09429d505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f5f0210a684ca2b9f5e4e09429d505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } } }
I20260620 12:29:14.885103 21644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:14.885059 21643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38f5f0210a684ca2b9f5e4e09429d505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f5f0210a684ca2b9f5e4e09429d505" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } } }
I20260620 12:29:14.885165 21643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:14.885453 21651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:14.888643 21651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:14.894501 21651 catalog_manager.cc:1347] Generated new cluster ID: 03350e033faa4ed0baff6933a9e2a8c3
I20260620 12:29:14.894577 21651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:14.910712 21651 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:14.911067 21651 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:14.922441 21651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38f5f0210a684ca2b9f5e4e09429d505: Generated new TSK 0
I20260620 12:29:14.923348 21651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:14.936097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21564
2026-06-20T12:29:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:14 dist-test-slave-bwpp krb5kdc[21663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:14 dist-test-slave-bwpp krb5kdc[21663](info): set up 2 sockets
Jun 20 12:29:14 dist-test-slave-bwpp krb5kdc[21663](info): commencing operation
krb5kdc: starting...
W20260620 12:29:15.971082 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:16 dist-test-slave-bwpp krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:16.097538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:16.210880 21676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:16.211133 21676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:16.211174 21676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:16.215337 21676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:16.215395 21676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:16.215415 21676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:16.215432 21676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:16.215447 21676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:16.220532 21676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:16.221457 21676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:16.222474 21676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:16.230487 21684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:16.230489 21681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:16.230507 21682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:16.230592 21676 server_base.cc:1034] running on GCE node
I20260620 12:29:16.231307 21676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:16.232332 21676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:16.233517 21676 hybrid_clock.cc:648] HybridClock initialized: now 1781958556233488 us; error 38 us; skew 500 ppm
Jun 20 12:29:16 dist-test-slave-bwpp krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:16.236259 21676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:16.237164 21676 webserver.cc:460] Webserver started at http://127.0.0.1:39687/ using document root <none> and password file <none>
I20260620 12:29:16.237704 21676 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:16.237793 21676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:16.237998 21676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:16.240002 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "276240f0c3a74b3fa729483d7184f523"
format_stamp: "Formatted at 2026-06-20 12:29:16 on dist-test-slave-bwpp"
server_key: "e6a8b3f3f387075f622389692ddc8c82"
server_key_iv: "f35e6442b93ae47a9166b37135fd9491"
server_key_version: "encryptionkey@0"
I20260620 12:29:16.240558 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "276240f0c3a74b3fa729483d7184f523"
format_stamp: "Formatted at 2026-06-20 12:29:16 on dist-test-slave-bwpp"
server_key: "e6a8b3f3f387075f622389692ddc8c82"
server_key_iv: "f35e6442b93ae47a9166b37135fd9491"
server_key_version: "encryptionkey@0"
I20260620 12:29:16.244078 21676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 12:29:16.246414 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:16.247686 21676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:16.247836 21676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "276240f0c3a74b3fa729483d7184f523"
format_stamp: "Formatted at 2026-06-20 12:29:16 on dist-test-slave-bwpp"
server_key: "e6a8b3f3f387075f622389692ddc8c82"
server_key_iv: "f35e6442b93ae47a9166b37135fd9491"
server_key_version: "encryptionkey@0"
I20260620 12:29:16.247949 21676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:16.284581 21676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:16.288147 21676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:16.288363 21676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:16.296310 21676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45791
I20260620 12:29:16.296360 21743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45791 every 8 connection(s)
I20260620 12:29:16.297576 21676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:16.300671 21744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:16.303580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21676
I20260620 12:29:16.303682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:16.303978 392 external_mini_cluster.cc:1428] Setting key cc8299d9d9ad2d754809a34307f6a6a8
I20260620 12:29:16.306916 21744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523: Bootstrap starting.
I20260620 12:29:16.309654 21744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:16.310636 21744 log.cc:824] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:16.313055 21744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523: No bootstrap required, opened a new log
Jun 20 12:29:16 dist-test-slave-bwpp krb5kdc[21663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:29:16.317113 21744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276240f0c3a74b3fa729483d7184f523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45791 } }
I20260620 12:29:16.317354 21744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:16.317427 21744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276240f0c3a74b3fa729483d7184f523, State: Initialized, Role: FOLLOWER
I20260620 12:29:16.317950 21744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276240f0c3a74b3fa729483d7184f523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45791 } }
I20260620 12:29:16.318137 21744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:16.318241 21744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:16.318357 21744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:16.319625 21744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276240f0c3a74b3fa729483d7184f523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45791 } }
I20260620 12:29:16.320003 21744 leader_election.cc:302] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276240f0c3a74b3fa729483d7184f523; no voters:
I20260620 12:29:16.320328 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:16.305518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:39687 (local address 127.0.0.1:45791)
0620 12:29:16.305958 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:16.305977 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:16.306020 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:29:16.306839 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:16.306859 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:16.307252 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:16.307501 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:16.308912 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:16.309993 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:16.311290 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:16.311958 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:16.312080 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:16.315137 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:16.315142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:16.315156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:16.315182 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:16.316949 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:16.317528 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:16.317533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:16.317538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:16.317648 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:16.318070 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:16.318072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:16.318074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:16.318615 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:16.318848 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:16.319162 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 12:29:16.319420 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":107,"threads_started":1}
I20260620 12:29:16.320348 21744 leader_election.cc:288] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:16.320458 21749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:16.320674 21749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [term 1 LEADER]: Becoming Leader. State: Replica: 276240f0c3a74b3fa729483d7184f523, State: Running, Role: LEADER
I20260620 12:29:16.320962 21749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276240f0c3a74b3fa729483d7184f523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45791 } }
I20260620 12:29:16.321811 21744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:16.321979 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "276240f0c3a74b3fa729483d7184f523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276240f0c3a74b3fa729483d7184f523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45791 } } }
I20260620 12:29:16.322109 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:16.322503 21753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:16.323889 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:16.325879 21753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:16.326155 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276240f0c3a74b3fa729483d7184f523. Latest consensus state: current_term: 1 leader_uuid: "276240f0c3a74b3fa729483d7184f523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276240f0c3a74b3fa729483d7184f523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45791 } } }
I20260620 12:29:16.326678 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:16.332605 21753 catalog_manager.cc:1347] Generated new cluster ID: 5407cb06f32f483da616ae366b087815
I20260620 12:29:16.332664 21753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:16.336298 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:16.326579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:47327 (local address 127.0.0.1:45791)
0620 12:29:16.326739 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:16.326744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:16.327271 (+ 527us) server_negotiation.cc:407] Connection header received
0620 12:29:16.327397 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:16.327400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:16.327448 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:16.327533 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:16.328422 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:16.329165 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:16.330439 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:16.330778 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:16.330895 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:16.332440 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:16.332443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:16.332445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:16.332464 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:16.334247 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:16.334920 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:16.334924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:16.334926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:16.335018 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:16.335380 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:16.335387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:16.335388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:16.335608 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:16.335782 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:16.336025 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 12:29:16.336129 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:29:16.344137 21753 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:16.344988 21753 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:16.354808 21753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 276240f0c3a74b3fa729483d7184f523: Generated new TSK 0
I20260620 12:29:16.355594 21753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:16.363893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21676
W20260620 12:29:16.369532 21661 connection.cc:535] client connection to 127.0.0.1:45791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45791): Connection reset by peer (error 104)
2026-06-20T12:29:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:16 dist-test-slave-bwpp krb5kdc[21775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:16 dist-test-slave-bwpp krb5kdc[21775](info): set up 2 sockets
Jun 20 12:29:16 dist-test-slave-bwpp krb5kdc[21775](info): commencing operation
krb5kdc: starting...
W20260620 12:29:17.418967 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:17 dist-test-slave-bwpp krb5kdc[21775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:17.537844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:17.643908 21788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:17.644191 21788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:17.644261 21788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:17.648205 21788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:17.648303 21788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:17.648342 21788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:17.648398 21788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:17.648437 21788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:17.653512 21788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:17.654500 21788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:17.655481 21788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:17.663683 21796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:17.663691 21794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:17.663795 21793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:17.663779 21788 server_base.cc:1034] running on GCE node
I20260620 12:29:17.664436 21788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:17.665616 21788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:17.666817 21788 hybrid_clock.cc:648] HybridClock initialized: now 1781958557666793 us; error 42 us; skew 500 ppm
Jun 20 12:29:17 dist-test-slave-bwpp krb5kdc[21775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:17.669798 21788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:17.670711 21788 webserver.cc:460] Webserver started at http://127.0.0.1:33801/ using document root <none> and password file <none>
I20260620 12:29:17.671259 21788 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:17.671320 21788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:17.671487 21788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:17.673594 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "d0dc1002a3b9484b8d8f86690eec060b"
format_stamp: "Formatted at 2026-06-20 12:29:17 on dist-test-slave-bwpp"
server_key: "1b72d6d81d29d96fc9a936673a09f5b8"
server_key_iv: "0b798bb8df396a63aed606b3572a7383"
server_key_version: "encryptionkey@0"
I20260620 12:29:17.674149 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0dc1002a3b9484b8d8f86690eec060b"
format_stamp: "Formatted at 2026-06-20 12:29:17 on dist-test-slave-bwpp"
server_key: "1b72d6d81d29d96fc9a936673a09f5b8"
server_key_iv: "0b798bb8df396a63aed606b3572a7383"
server_key_version: "encryptionkey@0"
I20260620 12:29:17.677925 21788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:29:17.680330 21803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:17.681506 21788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:29:17.681622 21788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "d0dc1002a3b9484b8d8f86690eec060b"
format_stamp: "Formatted at 2026-06-20 12:29:17 on dist-test-slave-bwpp"
server_key: "1b72d6d81d29d96fc9a936673a09f5b8"
server_key_iv: "0b798bb8df396a63aed606b3572a7383"
server_key_version: "encryptionkey@0"
I20260620 12:29:17.681710 21788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:17.724586 21788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:17.727722 21788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:17.727892 21788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:17.735491 21788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33289
I20260620 12:29:17.735512 21855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33289 every 8 connection(s)
I20260620 12:29:17.736500 21788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:17.739246 21856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:17.743557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21788
I20260620 12:29:17.743736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:17.744007 392 external_mini_cluster.cc:1428] Setting key 3158fcf23703f345e3831c4d1023df92
I20260620 12:29:17.744696 21856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b: Bootstrap starting.
I20260620 12:29:17.747023 21856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:17.747749 21856 log.cc:824] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:17.749505 21856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b: No bootstrap required, opened a new log
I20260620 12:29:17.752724 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:17.745376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:59673 (local address 127.0.0.1:33289)
0620 12:29:17.745794 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:17.745812 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:17.745846 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:29:17.746445 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:17.746460 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:17.746871 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:17.747190 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:17.747635 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:17.748630 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:17.749514 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:17.750035 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:17.750209 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:17.751104 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:17.751110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:17.751128 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:17.751521 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:17.751572 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:17.751599 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:17.751741 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:17.751935 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 12:29:17.752065 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":127,"threads_started":1}
I20260620 12:29:17.753228 21856 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dc1002a3b9484b8d8f86690eec060b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33289 } }
I20260620 12:29:17.753466 21856 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:17.753567 21856 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dc1002a3b9484b8d8f86690eec060b, State: Initialized, Role: FOLLOWER
I20260620 12:29:17.753968 21856 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dc1002a3b9484b8d8f86690eec060b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33289 } }
I20260620 12:29:17.754120 21856 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:17.754205 21856 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:17.754323 21856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:17.755446 21856 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dc1002a3b9484b8d8f86690eec060b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33289 } }
I20260620 12:29:17.755584 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:17.755792 21856 leader_election.cc:302] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0dc1002a3b9484b8d8f86690eec060b; no voters:
I20260620 12:29:17.756078 21856 leader_election.cc:288] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:17.756184 21861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:17.756388 21861 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [term 1 LEADER]: Becoming Leader. State: Replica: d0dc1002a3b9484b8d8f86690eec060b, State: Running, Role: LEADER
I20260620 12:29:17.756770 21861 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dc1002a3b9484b8d8f86690eec060b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33289 } }
I20260620 12:29:17.757256 21856 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:17.758420 21862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0dc1002a3b9484b8d8f86690eec060b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dc1002a3b9484b8d8f86690eec060b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33289 } } }
I20260620 12:29:17.758556 21862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:17.758852 21862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dc1002a3b9484b8d8f86690eec060b. Latest consensus state: current_term: 1 leader_uuid: "d0dc1002a3b9484b8d8f86690eec060b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dc1002a3b9484b8d8f86690eec060b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33289 } } }
I20260620 12:29:17.758929 21862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:17.762295 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:17.757711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:47579 (local address 127.0.0.1:33289)
0620 12:29:17.757803 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:17.757807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:17.758039 (+ 232us) server_negotiation.cc:407] Connection header received
0620 12:29:17.758230 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:17.758235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:17.758305 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:17.758364 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:17.758823 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:17.759559 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:17.760477 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:17.760813 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:17.760925 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:17.761823 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:17.761827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:17.761833 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:17.761849 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:17.761875 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:17.761887 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:17.761986 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:17.762142 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 12:29:17.762174 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 12:29:17.762348 21881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:17.762399 21881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:17.762553 21875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:17.763481 21875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:17.768838 21875 catalog_manager.cc:1347] Generated new cluster ID: bac6ab48d6284685955a66da938d2829
I20260620 12:29:17.768904 21875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:17.786553 21875 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:17.787514 21875 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:17.796440 21875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0dc1002a3b9484b8d8f86690eec060b: Generated new TSK 0
I20260620 12:29:17.797271 21875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:17.807919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21788
W20260620 12:29:17.813422 21773 connection.cc:535] client connection to 127.0.0.1:33289 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33289): Connection reset by peer (error 104)
2026-06-20T12:29:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:17 dist-test-slave-bwpp krb5kdc[21887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:17 dist-test-slave-bwpp krb5kdc[21887](info): set up 2 sockets
Jun 20 12:29:17 dist-test-slave-bwpp krb5kdc[21887](info): commencing operation
krb5kdc: starting...
W20260620 12:29:18.850994 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:18 dist-test-slave-bwpp krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:29:18.973577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:19.080458 21900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:19.080797 21900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:19.080881 21900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:19.084869 21900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:19.084975 21900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:19.085014 21900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:19.085052 21900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:19.085089 21900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:19.090283 21900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:19.091377 21900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:19.092422 21900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:19.100911 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:19.100919 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:19.100924 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:19.100998 21900 server_base.cc:1034] running on GCE node
I20260620 12:29:19.101712 21900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:19.102912 21900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:19.104075 21900 hybrid_clock.cc:648] HybridClock initialized: now 1781958559104061 us; error 25 us; skew 500 ppm
Jun 20 12:29:19 dist-test-slave-bwpp krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:19.106940 21900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:29:19.107823 21900 webserver.cc:460] Webserver started at http://127.0.0.1:40073/ using document root <none> and password file <none>
I20260620 12:29:19.108356 21900 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:19.108414 21900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:19.108642 21900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:19.110618 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "0b3f01fe878e433796e477c3a7762b97"
format_stamp: "Formatted at 2026-06-20 12:29:19 on dist-test-slave-bwpp"
server_key: "9141caf1815735e9aedfa615e0033095"
server_key_iv: "bc2504654d4954dcc6d33a2d17747568"
server_key_version: "encryptionkey@0"
I20260620 12:29:19.111224 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b3f01fe878e433796e477c3a7762b97"
format_stamp: "Formatted at 2026-06-20 12:29:19 on dist-test-slave-bwpp"
server_key: "9141caf1815735e9aedfa615e0033095"
server_key_iv: "bc2504654d4954dcc6d33a2d17747568"
server_key_version: "encryptionkey@0"
I20260620 12:29:19.114825 21900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:29:19.117247 21915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:19.118495 21900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:19.118623 21900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "0b3f01fe878e433796e477c3a7762b97"
format_stamp: "Formatted at 2026-06-20 12:29:19 on dist-test-slave-bwpp"
server_key: "9141caf1815735e9aedfa615e0033095"
server_key_iv: "bc2504654d4954dcc6d33a2d17747568"
server_key_version: "encryptionkey@0"
I20260620 12:29:19.118778 21900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:19.123145 21900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:19.126325 21900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:19.126516 21900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:19.133397 21900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34577
I20260620 12:29:19.133422 21967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34577 every 8 connection(s)
I20260620 12:29:19.134286 21900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:19.136947 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:19.139298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 21900
I20260620 12:29:19.139434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:19.139717 392 external_mini_cluster.cc:1428] Setting key bb6be0dbab7d1fc384f58c3fca291abf
I20260620 12:29:19.142966 21968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97: Bootstrap starting.
I20260620 12:29:19.144838 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:19.141120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:59601 (local address 127.0.0.1:34577)
0620 12:29:19.141553 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:19.141567 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:19.141601 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:29:19.142195 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:19.142211 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:19.142570 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:19.142961 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:19.143708 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:19.143713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:19.143731 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:19.143746 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:19.143797 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:19.143816 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:19.143964 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:19.144033 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 12:29:19.144183 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260620 12:29:19.145367 21968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:19.146093 21968 log.cc:824] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:19.147502 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:19.148047 21968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97: No bootstrap required, opened a new log
I20260620 12:29:19.151191 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:19.149510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:43649 (local address 127.0.0.1:34577)
0620 12:29:19.149615 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:19.149619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:19.149787 (+ 168us) server_negotiation.cc:407] Connection header received
0620 12:29:19.149891 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:19.149893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:19.149918 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:19.149988 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:19.150782 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:19.150784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:19.150786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:19.150796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:19.150819 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:19.150829 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:19.150901 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:19.151040 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 12:29:19.151070 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:29:19.152130 21968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3f01fe878e433796e477c3a7762b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260620 12:29:19.152359 21968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:19.152416 21968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3f01fe878e433796e477c3a7762b97, State: Initialized, Role: FOLLOWER
I20260620 12:29:19.152799 21968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3f01fe878e433796e477c3a7762b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260620 12:29:19.152946 21968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:19.153031 21968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:19.153127 21968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:19.153971 21968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3f01fe878e433796e477c3a7762b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260620 12:29:19.154299 21968 leader_election.cc:302] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b3f01fe878e433796e477c3a7762b97; no voters:
I20260620 12:29:19.154592 21968 leader_election.cc:288] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:19.154722 21978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:19.154968 21978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [term 1 LEADER]: Becoming Leader. State: Replica: 0b3f01fe878e433796e477c3a7762b97, State: Running, Role: LEADER
I20260620 12:29:19.155406 21978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3f01fe878e433796e477c3a7762b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } }
I20260620 12:29:19.155913 21968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:19.157122 21980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3f01fe878e433796e477c3a7762b97. Latest consensus state: current_term: 1 leader_uuid: "0b3f01fe878e433796e477c3a7762b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3f01fe878e433796e477c3a7762b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } } }
I20260620 12:29:19.157122 21979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b3f01fe878e433796e477c3a7762b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3f01fe878e433796e477c3a7762b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34577 } } }
I20260620 12:29:19.157272 21979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:19.157272 21980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:19.157754 21990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:19.160439 21990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:19.166042 21990 catalog_manager.cc:1347] Generated new cluster ID: 1fec555409d44c5989e53440774bfdc6
I20260620 12:29:19.166112 21990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:19.179931 21990 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:19.180306 21990 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:19.186205 21990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b3f01fe878e433796e477c3a7762b97: Generated new TSK 0
I20260620 12:29:19.187088 21990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:19.196298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 21900
2026-06-20T12:29:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:19 dist-test-slave-bwpp krb5kdc[21999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:19 dist-test-slave-bwpp krb5kdc[21999](info): set up 2 sockets
Jun 20 12:29:19 dist-test-slave-bwpp krb5kdc[21999](info): commencing operation
krb5kdc: starting...
W20260620 12:29:20.242918 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:20 dist-test-slave-bwpp krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:20.361269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:20.470396 22012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:20.470722 22012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:20.470786 22012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:20.474745 22012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:20.474803 22012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:20.474822 22012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:20.474839 22012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:20.474853 22012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:20.480422 22012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:20.481453 22012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:20.482504 22012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:20.490952 22017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:20.490926 22020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:20.491289 22012 server_base.cc:1034] running on GCE node
W20260620 12:29:20.491390 22018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:20.491905 22012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:20.493009 22012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:20.494207 22012 hybrid_clock.cc:648] HybridClock initialized: now 1781958560494185 us; error 36 us; skew 500 ppm
Jun 20 12:29:20 dist-test-slave-bwpp krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:20.502393 22012 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:20.503350 22012 webserver.cc:460] Webserver started at http://127.0.98.62:44405/ using document root <none> and password file <none>
I20260620 12:29:20.503883 22012 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:20.503944 22012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:20.504117 22012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:20.506142 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "de9d740ae89c4f329216049496112834"
format_stamp: "Formatted at 2026-06-20 12:29:20 on dist-test-slave-bwpp"
server_key: "28dce6cd0bc20f17480c48cc6cd74ad7"
server_key_iv: "2ed60e6798e80ae51957496b2f3245cc"
server_key_version: "encryptionkey@0"
I20260620 12:29:20.506726 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "de9d740ae89c4f329216049496112834"
format_stamp: "Formatted at 2026-06-20 12:29:20 on dist-test-slave-bwpp"
server_key: "28dce6cd0bc20f17480c48cc6cd74ad7"
server_key_iv: "2ed60e6798e80ae51957496b2f3245cc"
server_key_version: "encryptionkey@0"
I20260620 12:29:20.510318 22012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 12:29:20.512756 22027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:20.513973 22012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:20.514083 22012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "de9d740ae89c4f329216049496112834"
format_stamp: "Formatted at 2026-06-20 12:29:20 on dist-test-slave-bwpp"
server_key: "28dce6cd0bc20f17480c48cc6cd74ad7"
server_key_iv: "2ed60e6798e80ae51957496b2f3245cc"
server_key_version: "encryptionkey@0"
I20260620 12:29:20.514166 22012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:20.529693 22012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:20.533042 22012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:20.533195 22012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:20.540566 22012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37813
I20260620 12:29:20.540593 22079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37813 every 8 connection(s)
I20260620 12:29:20.541523 22012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:20.544312 22080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:20.546828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22012
I20260620 12:29:20.546958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:20.547235 392 external_mini_cluster.cc:1428] Setting key 02f6cce721e8253d622662e646fd60fd
I20260620 12:29:20.550700 22080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834: Bootstrap starting.
I20260620 12:29:20.553289 22080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:20.554069 22080 log.cc:824] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:20.555992 22080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834: No bootstrap required, opened a new log
Jun 20 12:29:20 dist-test-slave-bwpp krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:20.559989 22080 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9d740ae89c4f329216049496112834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 12:29:20.560328 22080 raft_consensus.cc:374] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:20.560390 22080 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9d740ae89c4f329216049496112834, State: Initialized, Role: FOLLOWER
I20260620 12:29:20.560894 22080 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9d740ae89c4f329216049496112834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 12:29:20.561080 22080 raft_consensus.cc:388] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:20.561153 22080 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:20.561257 22080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:20.562078 22080 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9d740ae89c4f329216049496112834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 12:29:20.562463 22080 leader_election.cc:302] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de9d740ae89c4f329216049496112834; no voters:
I20260620 12:29:20.562867 22080 leader_election.cc:288] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:20.562932 22085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:20.563113 22085 raft_consensus.cc:686] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [term 1 LEADER]: Becoming Leader. State: Replica: de9d740ae89c4f329216049496112834, State: Running, Role: LEADER
I20260620 12:29:20.563401 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:20.548684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53873 (local address 127.0.98.62:37813)
0620 12:29:20.549113 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:20.549131 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:20.549174 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:29:20.550020 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:20.550043 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:20.550458 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:20.550845 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:20.551530 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:20.552985 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:20.553894 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:20.554565 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:20.554764 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:20.557314 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:20.557320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:20.557336 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:20.557367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:20.560005 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:20.560629 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:20.560634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:20.560639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:20.560782 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:20.561123 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:20.561128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:20.561130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:20.561628 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:20.561871 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:20.562198 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 12:29:20.562421 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":103,"threads_started":1}
I20260620 12:29:20.563447 22085 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9d740ae89c4f329216049496112834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 12:29:20.564102 22080 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:20.565294 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de9d740ae89c4f329216049496112834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9d740ae89c4f329216049496112834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } } }
I20260620 12:29:20.565418 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:20.565779 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9d740ae89c4f329216049496112834. Latest consensus state: current_term: 1 leader_uuid: "de9d740ae89c4f329216049496112834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9d740ae89c4f329216049496112834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } } }
I20260620 12:29:20.565939 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:20.566243 22094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:20.566694 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:20.569387 22094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:20.575757 22094 catalog_manager.cc:1347] Generated new cluster ID: 708e394725514bd1880ac8e728897a46
I20260620 12:29:20.575899 22094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:20.578094 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:20.568849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53929 (local address 127.0.98.62:37813)
0620 12:29:20.568969 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:20.568973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:20.569234 (+ 261us) server_negotiation.cc:407] Connection header received
0620 12:29:20.569368 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:20.569371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:20.569422 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:20.569513 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:20.570373 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:20.571207 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:20.572091 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:20.572354 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:20.572473 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:20.574098 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:20.574102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:20.574105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:20.574133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:20.575846 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:20.576425 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:20.576428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:20.576430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:20.576531 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:20.576953 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:20.576957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:20.576959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:20.577148 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:20.577278 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:20.577861 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 12:29:20.577947 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 12:29:20.594761 22094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:20.596069 22094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:20.609671 22094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de9d740ae89c4f329216049496112834: Generated new TSK 0
I20260620 12:29:20.610602 22094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:20.616133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22012
W20260620 12:29:20.621500 21997 connection.cc:535] client connection to 127.0.98.62:37813 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37813): Connection reset by peer (error 104)
2026-06-20T12:29:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:20 dist-test-slave-bwpp krb5kdc[22111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:20 dist-test-slave-bwpp krb5kdc[22111](info): set up 2 sockets
Jun 20 12:29:20 dist-test-slave-bwpp krb5kdc[22111](info): commencing operation
krb5kdc: starting...
W20260620 12:29:21.646979 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:21 dist-test-slave-bwpp krb5kdc[22111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:21.763543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:21.874488 22124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:21.874822 22124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:21.874867 22124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:21.878904 22124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:21.878961 22124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:21.878981 22124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:21.878997 22124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:21.879012 22124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:21.884174 22124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:21.885171 22124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:21.886173 22124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:21.894870 22129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:21.894891 22132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:21.894896 22130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:21.895179 22124 server_base.cc:1034] running on GCE node
I20260620 12:29:21.895747 22124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:21.896837 22124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:21.898039 22124 hybrid_clock.cc:648] HybridClock initialized: now 1781958561898019 us; error 33 us; skew 500 ppm
Jun 20 12:29:21 dist-test-slave-bwpp krb5kdc[22111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:21.900933 22124 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:21.901801 22124 webserver.cc:460] Webserver started at http://127.0.98.62:43845/ using document root <none> and password file <none>
I20260620 12:29:21.902328 22124 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:21.902387 22124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:21.902544 22124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:21.904618 22124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "16fa094883b544c780ce455cf946c557"
format_stamp: "Formatted at 2026-06-20 12:29:21 on dist-test-slave-bwpp"
server_key: "1e4e666df329b70e469a25e05fe20709"
server_key_iv: "1a4c4f30a40982dc8a32a9068dd98a97"
server_key_version: "encryptionkey@0"
I20260620 12:29:21.905143 22124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "16fa094883b544c780ce455cf946c557"
format_stamp: "Formatted at 2026-06-20 12:29:21 on dist-test-slave-bwpp"
server_key: "1e4e666df329b70e469a25e05fe20709"
server_key_iv: "1a4c4f30a40982dc8a32a9068dd98a97"
server_key_version: "encryptionkey@0"
I20260620 12:29:21.908906 22124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:29:21.911404 22139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:21.912617 22124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:21.912761 22124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "16fa094883b544c780ce455cf946c557"
format_stamp: "Formatted at 2026-06-20 12:29:21 on dist-test-slave-bwpp"
server_key: "1e4e666df329b70e469a25e05fe20709"
server_key_iv: "1a4c4f30a40982dc8a32a9068dd98a97"
server_key_version: "encryptionkey@0"
I20260620 12:29:21.912868 22124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:21.926550 22124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:21.929872 22124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:21.930102 22124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:21.937137 22124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46195
I20260620 12:29:21.937216 22191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46195 every 8 connection(s)
I20260620 12:29:21.938125 22124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:21.939689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22124
I20260620 12:29:21.939865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:21.940158 392 external_mini_cluster.cc:1428] Setting key 34644c47d9039d246cb00fca75c82d23
I20260620 12:29:21.941109 22192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:21.947952 22192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557: Bootstrap starting.
I20260620 12:29:21.949879 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:21.941575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49941 (local address 127.0.98.62:46195)
0620 12:29:21.942039 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:21.942055 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:21.942397 (+ 342us) server_negotiation.cc:407] Connection header received
0620 12:29:21.943247 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:21.943263 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:21.943593 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:21.943878 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:21.944324 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:21.945326 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:21.946542 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:21.947089 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:21.947203 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:21.948193 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:21.948198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:21.948210 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:21.948590 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:21.948640 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:21.948662 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:21.948773 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:21.948990 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 12:29:21.949139 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":142,"threads_started":1}
I20260620 12:29:21.950383 22192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:21.951189 22192 log.cc:824] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:21.952979 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:21.953279 22192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557: No bootstrap required, opened a new log
I20260620 12:29:21.957587 22192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa094883b544c780ce455cf946c557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260620 12:29:21.957870 22192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:21.957942 22192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fa094883b544c780ce455cf946c557, State: Initialized, Role: FOLLOWER
I20260620 12:29:21.958441 22192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa094883b544c780ce455cf946c557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260620 12:29:21.958617 22192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:21.958757 22192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:21.958868 22192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:21.959214 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:21.954966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55575 (local address 127.0.98.62:46195)
0620 12:29:21.955093 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:21.955097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:21.955309 (+ 212us) server_negotiation.cc:407] Connection header received
0620 12:29:21.955433 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:21.955435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:21.955471 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:21.955533 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:21.955851 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:21.956355 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:21.957454 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:21.957680 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:21.957763 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:21.958695 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:21.958701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:21.958704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:21.958717 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:21.958747 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:21.958763 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:21.958878 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:21.959026 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 12:29:21.959058 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:29:21.959964 22192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa094883b544c780ce455cf946c557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260620 12:29:21.960331 22192 leader_election.cc:302] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16fa094883b544c780ce455cf946c557; no voters:
I20260620 12:29:21.960606 22192 leader_election.cc:288] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:21.960736 22202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:21.961021 22202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [term 1 LEADER]: Becoming Leader. State: Replica: 16fa094883b544c780ce455cf946c557, State: Running, Role: LEADER
I20260620 12:29:21.961380 22202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa094883b544c780ce455cf946c557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260620 12:29:21.961841 22192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:21.963083 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16fa094883b544c780ce455cf946c557. Latest consensus state: current_term: 1 leader_uuid: "16fa094883b544c780ce455cf946c557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa094883b544c780ce455cf946c557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } } }
I20260620 12:29:21.963222 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:21.963533 22203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16fa094883b544c780ce455cf946c557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa094883b544c780ce455cf946c557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } } }
I20260620 12:29:21.963608 22211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:21.963708 22203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:21.966929 22211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:21.973147 22211 catalog_manager.cc:1347] Generated new cluster ID: 623c715c21164ebb86632e1f8efe6424
I20260620 12:29:21.973325 22211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:21.992967 22211 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:21.994089 22211 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:22.001276 22211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16fa094883b544c780ce455cf946c557: Generated new TSK 0
I20260620 12:29:22.002080 22211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:22.009429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22124
W20260620 12:29:22.014817 22109 connection.cc:535] client connection to 127.0.98.62:46195 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46195): Connection reset by peer (error 104)
2026-06-20T12:29:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:22 dist-test-slave-bwpp krb5kdc[22223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:22 dist-test-slave-bwpp krb5kdc[22223](info): set up 2 sockets
Jun 20 12:29:22 dist-test-slave-bwpp krb5kdc[22223](info): commencing operation
krb5kdc: starting...
W20260620 12:29:23.062997 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:23 dist-test-slave-bwpp krb5kdc[22223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:23.185860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43967
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:23.292223 22236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:23.292470 22236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:23.292510 22236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:23.296330 22236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:23.296378 22236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:23.296396 22236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:23.296412 22236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:23.296427 22236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:23.301497 22236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:23.302430 22236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:23.303368 22236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:29:23.311736 22236 server_base.cc:1034] running on GCE node
W20260620 12:29:23.311666 22242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:23.311702 22244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:23.311664 22241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:23.312605 22236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:23.313668 22236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:23.314863 22236 hybrid_clock.cc:648] HybridClock initialized: now 1781958563314837 us; error 38 us; skew 500 ppm
Jun 20 12:29:23 dist-test-slave-bwpp krb5kdc[22223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:23.317688 22236 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:23.318552 22236 webserver.cc:460] Webserver started at http://127.0.98.62:45275/ using document root <none> and password file <none>
I20260620 12:29:23.319157 22236 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:23.319255 22236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:23.319485 22236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:23.321558 22236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "06d6aa5a8a6e470e8d3023fac3e652cc"
format_stamp: "Formatted at 2026-06-20 12:29:23 on dist-test-slave-bwpp"
server_key: "c6f2a716b1c07db92e56440c44602da1"
server_key_iv: "ddb7ea2fb9a30e286e27b67ee8e2d87c"
server_key_version: "encryptionkey@0"
I20260620 12:29:23.322124 22236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "06d6aa5a8a6e470e8d3023fac3e652cc"
format_stamp: "Formatted at 2026-06-20 12:29:23 on dist-test-slave-bwpp"
server_key: "c6f2a716b1c07db92e56440c44602da1"
server_key_iv: "ddb7ea2fb9a30e286e27b67ee8e2d87c"
server_key_version: "encryptionkey@0"
I20260620 12:29:23.325783 22236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:29:23.328260 22251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:23.329613 22236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:29:23.329823 22236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "06d6aa5a8a6e470e8d3023fac3e652cc"
format_stamp: "Formatted at 2026-06-20 12:29:23 on dist-test-slave-bwpp"
server_key: "c6f2a716b1c07db92e56440c44602da1"
server_key_iv: "ddb7ea2fb9a30e286e27b67ee8e2d87c"
server_key_version: "encryptionkey@0"
I20260620 12:29:23.330003 22236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:23.334599 22236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:23.337566 22236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:23.337762 22236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:23.344723 22236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43967
I20260620 12:29:23.344748 22303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43967 every 8 connection(s)
I20260620 12:29:23.345665 22236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:23.348287 22304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:23.351167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22236
I20260620 12:29:23.351274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:23.351599 392 external_mini_cluster.cc:1428] Setting key ecd88d3c9bea5793047c6e266e4a078b
I20260620 12:29:23.354012 22304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc: Bootstrap starting.
I20260620 12:29:23.356626 22304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:23.356986 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:23.352949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54609 (local address 127.0.98.62:43967)
0620 12:29:23.353384 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:23.353402 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:23.353441 (+ 39us) server_negotiation.cc:407] Connection header received
0620 12:29:23.354238 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:23.354259 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:23.354601 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:23.354969 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:23.355681 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:23.355686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:23.355699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:23.355710 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:23.355767 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:23.355792 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:23.355919 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:23.355984 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 12:29:23.356119 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260620 12:29:23.357438 22304 log.cc:824] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:23.359264 22304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc: No bootstrap required, opened a new log
I20260620 12:29:23.359628 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:23.363546 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:23.361788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42153 (local address 127.0.98.62:43967)
0620 12:29:23.361911 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:23.361915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:23.362156 (+ 241us) server_negotiation.cc:407] Connection header received
0620 12:29:23.362272 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:23.362274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:23.362304 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:23.362367 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:23.363154 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:23.363156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:23.363158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:23.363168 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:23.363191 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:23.363201 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:23.363268 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:23.363374 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 12:29:23.363404 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:29:23.363963 22304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43967 } }
I20260620 12:29:23.364187 22304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:23.364253 22304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d6aa5a8a6e470e8d3023fac3e652cc, State: Initialized, Role: FOLLOWER
I20260620 12:29:23.364698 22304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43967 } }
I20260620 12:29:23.364827 22304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:23.364912 22304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:23.365000 22304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:23.365844 22304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43967 } }
I20260620 12:29:23.366171 22304 leader_election.cc:302] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d6aa5a8a6e470e8d3023fac3e652cc; no voters:
I20260620 12:29:23.366462 22304 leader_election.cc:288] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:23.366582 22314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:23.366828 22314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [term 1 LEADER]: Becoming Leader. State: Replica: 06d6aa5a8a6e470e8d3023fac3e652cc, State: Running, Role: LEADER
I20260620 12:29:23.367118 22314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43967 } }
I20260620 12:29:23.367643 22304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:23.368245 22314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d6aa5a8a6e470e8d3023fac3e652cc. Latest consensus state: current_term: 1 leader_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43967 } } }
I20260620 12:29:23.368427 22314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:23.368819 22322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:23.368971 22316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d6aa5a8a6e470e8d3023fac3e652cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43967 } } }
I20260620 12:29:23.369128 22316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:23.372232 22322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:23.377480 22322 catalog_manager.cc:1347] Generated new cluster ID: 8622728871ce4c51ac5ee07e8d5e82ed
I20260620 12:29:23.377542 22322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:23.386541 22322 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:23.386879 22322 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:23.393586 22322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d6aa5a8a6e470e8d3023fac3e652cc: Generated new TSK 0
I20260620 12:29:23.394315 22322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:23.398211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22236
2026-06-20T12:29:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:23 dist-test-slave-bwpp krb5kdc[22335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:23 dist-test-slave-bwpp krb5kdc[22335](info): set up 2 sockets
Jun 20 12:29:23 dist-test-slave-bwpp krb5kdc[22335](info): commencing operation
krb5kdc: starting...
W20260620 12:29:24.431082 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:24 dist-test-slave-bwpp krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:24.551775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:24.661558 22348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:24.661872 22348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:24.661967 22348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:24.666215 22348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:24.666302 22348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:24.666344 22348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:24.666399 22348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:24.666435 22348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:24.671613 22348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:24.672612 22348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:24.673611 22348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:24.682462 22353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:24.682332 22356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:24.682415 22354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:24.682557 22348 server_base.cc:1034] running on GCE node
I20260620 12:29:24.683224 22348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:24.684278 22348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:24.685464 22348 hybrid_clock.cc:648] HybridClock initialized: now 1781958564685433 us; error 43 us; skew 500 ppm
Jun 20 12:29:24 dist-test-slave-bwpp krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:24.688210 22348 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:24.689056 22348 webserver.cc:460] Webserver started at http://127.0.98.62:33355/ using document root <none> and password file <none>
I20260620 12:29:24.689659 22348 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:24.689721 22348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:24.690024 22348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:24.692087 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "58e5aa5f39f447f4956ad8b2a1f4a728"
format_stamp: "Formatted at 2026-06-20 12:29:24 on dist-test-slave-bwpp"
server_key: "b188460606c264ae35856799b8a2eedd"
server_key_iv: "aed5db95b3770271eba06694ec18daf6"
server_key_version: "encryptionkey@0"
I20260620 12:29:24.692641 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "58e5aa5f39f447f4956ad8b2a1f4a728"
format_stamp: "Formatted at 2026-06-20 12:29:24 on dist-test-slave-bwpp"
server_key: "b188460606c264ae35856799b8a2eedd"
server_key_iv: "aed5db95b3770271eba06694ec18daf6"
server_key_version: "encryptionkey@0"
I20260620 12:29:24.696273 22348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:29:24.698549 22363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:24.699743 22348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:24.699879 22348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "58e5aa5f39f447f4956ad8b2a1f4a728"
format_stamp: "Formatted at 2026-06-20 12:29:24 on dist-test-slave-bwpp"
server_key: "b188460606c264ae35856799b8a2eedd"
server_key_iv: "aed5db95b3770271eba06694ec18daf6"
server_key_version: "encryptionkey@0"
I20260620 12:29:24.699980 22348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:24.736642 22348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:24.739708 22348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:24.739925 22348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:24.746562 22348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35229
I20260620 12:29:24.746649 22415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35229 every 8 connection(s)
I20260620 12:29:24.747540 22348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:24.748039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22348
I20260620 12:29:24.748190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:24.748477 392 external_mini_cluster.cc:1428] Setting key 9ba26c2c2ce84e841faf4db39288c4f7
I20260620 12:29:24.750713 22416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:24.756042 22416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728: Bootstrap starting.
Jun 20 12:29:24 dist-test-slave-bwpp krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:24.758302 22416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:24.759119 22416 log.cc:824] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:24.761271 22416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728: No bootstrap required, opened a new log
I20260620 12:29:24.762166 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:24.749829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44459 (local address 127.0.98.62:35229)
0620 12:29:24.750263 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:24.750277 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:24.750313 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:29:24.751068 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:24.751085 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:24.751414 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:24.751748 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:24.752556 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:24.753553 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:24.754419 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:24.754972 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:24.755143 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:24.757619 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:24.757624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:24.757636 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:24.757663 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:24.759460 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:24.759937 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:24.759942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:24.759946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:24.760050 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:24.760337 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:24.760339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:24.760340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:24.760695 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:24.760835 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:24.761174 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 12:29:24.761464 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260620 12:29:24.764438 22416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260620 12:29:24.764708 22416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:24.764775 22416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e5aa5f39f447f4956ad8b2a1f4a728, State: Initialized, Role: FOLLOWER
I20260620 12:29:24.765209 22416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260620 12:29:24.765368 22416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:24.765483 22416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:24.765525 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:24.765594 22416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:24.766717 22416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260620 12:29:24.767098 22416 leader_election.cc:302] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e5aa5f39f447f4956ad8b2a1f4a728; no voters:
I20260620 12:29:24.767457 22416 leader_election.cc:288] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:24.767575 22425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:24.767830 22425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [term 1 LEADER]: Becoming Leader. State: Replica: 58e5aa5f39f447f4956ad8b2a1f4a728, State: Running, Role: LEADER
I20260620 12:29:24.768240 22425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260620 12:29:24.768848 22416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:24.769531 22425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e5aa5f39f447f4956ad8b2a1f4a728. Latest consensus state: current_term: 1 leader_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } } }
I20260620 12:29:24.770277 22425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:24.770169 22426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e5aa5f39f447f4956ad8b2a1f4a728" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } } }
I20260620 12:29:24.770562 22426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:24.770874 22437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:24.774149 22437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:24.776599 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:24.768132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35463 (local address 127.0.98.62:35229)
0620 12:29:24.768234 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:24.768238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:24.768253 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:29:24.768430 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:24.768433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:24.768478 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:24.768561 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:24.769352 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:24.770213 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:24.771701 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:24.771947 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:24.772061 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:24.773417 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:24.773420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:24.773423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:24.773447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:24.774955 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:24.775387 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:24.775392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:24.775394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:24.775473 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:24.775752 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:24.775755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:24.775757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:24.775929 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:24.776064 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:24.776303 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 12:29:24.776421 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:29:24.780153 22437 catalog_manager.cc:1347] Generated new cluster ID: a36c19c3fee842c68669d4b2a957d87a
I20260620 12:29:24.780223 22437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:24.798341 22437 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:24.799353 22437 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:24.818507 22437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e5aa5f39f447f4956ad8b2a1f4a728: Generated new TSK 0
I20260620 12:29:24.819294 22437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:24.825227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22348
2026-06-20T12:29:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:24 dist-test-slave-bwpp krb5kdc[22447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:24 dist-test-slave-bwpp krb5kdc[22447](info): set up 2 sockets
Jun 20 12:29:24 dist-test-slave-bwpp krb5kdc[22447](info): commencing operation
krb5kdc: starting...
W20260620 12:29:25.868640 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:25 dist-test-slave-bwpp krb5kdc[22447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:25.991482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:26.103672 22460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:26.103917 22460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:26.103957 22460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:26.107854 22460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:26.107904 22460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:26.107923 22460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:26.107939 22460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:26.107954 22460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:26.112903 22460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:26.113807 22460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:26.114816 22460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:26.122860 22465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:26.122857 22468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:26.122895 22466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:26.123191 22460 server_base.cc:1034] running on GCE node
I20260620 12:29:26.123723 22460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:26.124789 22460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:26.125980 22460 hybrid_clock.cc:648] HybridClock initialized: now 1781958566125952 us; error 41 us; skew 500 ppm
Jun 20 12:29:26 dist-test-slave-bwpp krb5kdc[22447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:26.133819 22460 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:26.134873 22460 webserver.cc:460] Webserver started at http://127.0.98.62:38149/ using document root <none> and password file <none>
I20260620 12:29:26.135423 22460 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:26.135516 22460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:26.135720 22460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:26.137733 22460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "4c88f2ba17814f9fbd489995ae75290d"
format_stamp: "Formatted at 2026-06-20 12:29:26 on dist-test-slave-bwpp"
server_key: "ec2a227ae891dd80d82c35b818a446bb"
server_key_iv: "dac115b8227a249828f7662b0ce4da82"
server_key_version: "encryptionkey@0"
I20260620 12:29:26.138304 22460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c88f2ba17814f9fbd489995ae75290d"
format_stamp: "Formatted at 2026-06-20 12:29:26 on dist-test-slave-bwpp"
server_key: "ec2a227ae891dd80d82c35b818a446bb"
server_key_iv: "dac115b8227a249828f7662b0ce4da82"
server_key_version: "encryptionkey@0"
I20260620 12:29:26.141849 22460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:29:26.145234 22475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:26.146911 22460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 12:29:26.147033 22460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "4c88f2ba17814f9fbd489995ae75290d"
format_stamp: "Formatted at 2026-06-20 12:29:26 on dist-test-slave-bwpp"
server_key: "ec2a227ae891dd80d82c35b818a446bb"
server_key_iv: "dac115b8227a249828f7662b0ce4da82"
server_key_version: "encryptionkey@0"
I20260620 12:29:26.147141 22460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:26.173327 22460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:26.176422 22460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:26.176601 22460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:26.185216 22460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34593
I20260620 12:29:26.185217 22527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34593 every 8 connection(s)
I20260620 12:29:26.186280 22460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:26.187230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22460
I20260620 12:29:26.187371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:26.187703 392 external_mini_cluster.cc:1428] Setting key c6000850c2bbf7aaf2061f92328e6c91
I20260620 12:29:26.189852 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:26.195240 22528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d: Bootstrap starting.
I20260620 12:29:26.196597 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:26.189288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39591 (local address 127.0.98.62:34593)
0620 12:29:26.189691 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:26.189713 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:26.189747 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:29:26.190428 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:26.190445 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:26.190818 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:26.191145 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:26.191577 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:26.192542 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:26.193426 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:26.193898 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:26.194068 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:26.194958 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:26.194963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:26.194975 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:26.195392 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:26.195444 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:26.195465 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:26.195571 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:26.195696 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 12:29:26.195923 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":141,"threads_started":1}
I20260620 12:29:26.197353 22528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:26.198036 22528 log.cc:824] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:26.199291 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:26.200320 22528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d: No bootstrap required, opened a new log
I20260620 12:29:26.203908 22528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c88f2ba17814f9fbd489995ae75290d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34593 } }
I20260620 12:29:26.204185 22528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:26.204229 22528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c88f2ba17814f9fbd489995ae75290d, State: Initialized, Role: FOLLOWER
I20260620 12:29:26.204731 22528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c88f2ba17814f9fbd489995ae75290d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34593 } }
I20260620 12:29:26.204931 22528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:26.205005 22528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:26.205113 22528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:26.206056 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:26.201487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60451 (local address 127.0.98.62:34593)
0620 12:29:26.201613 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:26.201617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:26.201798 (+ 181us) server_negotiation.cc:407] Connection header received
0620 12:29:26.201902 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:26.201905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:26.201940 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:26.202002 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:26.202294 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:26.203369 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:26.204448 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:26.204707 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:26.204804 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:26.205652 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:26.205655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:26.205657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:26.205667 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:26.205689 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:26.205698 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:26.205759 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:26.205868 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 12:29:26.205910 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:29:26.206418 22528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c88f2ba17814f9fbd489995ae75290d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34593 } }
I20260620 12:29:26.206813 22528 leader_election.cc:302] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c88f2ba17814f9fbd489995ae75290d; no voters:
I20260620 12:29:26.207108 22528 leader_election.cc:288] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:26.207218 22538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:26.207484 22538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [term 1 LEADER]: Becoming Leader. State: Replica: 4c88f2ba17814f9fbd489995ae75290d, State: Running, Role: LEADER
I20260620 12:29:26.207790 22538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c88f2ba17814f9fbd489995ae75290d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34593 } }
I20260620 12:29:26.208287 22528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:26.209302 22540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c88f2ba17814f9fbd489995ae75290d. Latest consensus state: current_term: 1 leader_uuid: "4c88f2ba17814f9fbd489995ae75290d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c88f2ba17814f9fbd489995ae75290d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34593 } } }
I20260620 12:29:26.209424 22540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:26.209683 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c88f2ba17814f9fbd489995ae75290d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c88f2ba17814f9fbd489995ae75290d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34593 } } }
I20260620 12:29:26.209772 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:26.209882 22547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:26.212956 22547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:26.218742 22547 catalog_manager.cc:1347] Generated new cluster ID: 27faf8e4ac0c48dc89a50b31f2d2b652
I20260620 12:29:26.218809 22547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:26.232788 22547 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:26.233629 22547 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:26.244828 22547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c88f2ba17814f9fbd489995ae75290d: Generated new TSK 0
I20260620 12:29:26.245528 22547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:26.249774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22460
2026-06-20T12:29:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:26 dist-test-slave-bwpp krb5kdc[22559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:26 dist-test-slave-bwpp krb5kdc[22559](info): set up 2 sockets
Jun 20 12:29:26 dist-test-slave-bwpp krb5kdc[22559](info): commencing operation
krb5kdc: starting...
W20260620 12:29:27.302925 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:27 dist-test-slave-bwpp krb5kdc[22559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:27.421407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33261
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:27.528884 22572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:27.529176 22572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:27.529258 22572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:27.533258 22572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:27.533344 22572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:27.533385 22572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:27.533465 22572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:27.533505 22572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:27.538609 22572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:27.539621 22572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:27.540581 22572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:27.548831 22577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:27.548825 22580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:27.548827 22578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:27.549093 22572 server_base.cc:1034] running on GCE node
I20260620 12:29:27.549656 22572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:27.550719 22572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:27.551883 22572 hybrid_clock.cc:648] HybridClock initialized: now 1781958567551856 us; error 41 us; skew 500 ppm
Jun 20 12:29:27 dist-test-slave-bwpp krb5kdc[22559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:27.554497 22572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:27.555343 22572 webserver.cc:460] Webserver started at http://127.0.98.62:36887/ using document root <none> and password file <none>
I20260620 12:29:27.555850 22572 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:27.555908 22572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:27.556063 22572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:27.557998 22572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "91d19dd9e2d44227a7c2dd035c49f738"
format_stamp: "Formatted at 2026-06-20 12:29:27 on dist-test-slave-bwpp"
server_key: "40df20c1bfb483ac455306279d9df9b5"
server_key_iv: "d8218346c775f0c0a757c621eeaa7923"
server_key_version: "encryptionkey@0"
I20260620 12:29:27.558581 22572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "91d19dd9e2d44227a7c2dd035c49f738"
format_stamp: "Formatted at 2026-06-20 12:29:27 on dist-test-slave-bwpp"
server_key: "40df20c1bfb483ac455306279d9df9b5"
server_key_iv: "d8218346c775f0c0a757c621eeaa7923"
server_key_version: "encryptionkey@0"
I20260620 12:29:27.562094 22572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:29:27.564507 22587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:27.565685 22572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:27.565855 22572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "91d19dd9e2d44227a7c2dd035c49f738"
format_stamp: "Formatted at 2026-06-20 12:29:27 on dist-test-slave-bwpp"
server_key: "40df20c1bfb483ac455306279d9df9b5"
server_key_iv: "d8218346c775f0c0a757c621eeaa7923"
server_key_version: "encryptionkey@0"
I20260620 12:29:27.565971 22572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:27.570453 22572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:27.573114 22572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:27.573310 22572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:27.580379 22572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33261
I20260620 12:29:27.580444 22639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33261 every 8 connection(s)
I20260620 12:29:27.581314 22572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:27.584002 22640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:27.586802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22572
I20260620 12:29:27.586956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:27.587231 392 external_mini_cluster.cc:1428] Setting key 6af50aeb959ea9866f792c0db7b7d39f
I20260620 12:29:27.589632 22640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738: Bootstrap starting.
I20260620 12:29:27.591998 22640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:27.592202 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:27.588512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58693 (local address 127.0.98.62:33261)
0620 12:29:27.588923 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:27.588939 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:27.588971 (+ 32us) server_negotiation.cc:407] Connection header received
0620 12:29:27.589555 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:27.589569 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:27.589898 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:27.590237 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:27.591071 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:27.591076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:27.591086 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:27.591097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:27.591144 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:27.591161 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:27.591309 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:27.591377 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:29:27.591524 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260620 12:29:27.592785 22640 log.cc:824] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:27.594775 22640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738: No bootstrap required, opened a new log
I20260620 12:29:27.594924 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:27.598773 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:27.597023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59609 (local address 127.0.98.62:33261)
0620 12:29:27.597164 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:27.597167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:27.597313 (+ 146us) server_negotiation.cc:407] Connection header received
0620 12:29:27.597402 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:27.597405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:27.597435 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:27.597508 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:27.598281 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:27.598284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:27.598286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:29:27.598297 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:27.598322 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:29:27.598332 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:27.598407 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:27.598529 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 12:29:27.598563 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 12:29:27.599112 22640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d19dd9e2d44227a7c2dd035c49f738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33261 } }
I20260620 12:29:27.599355 22640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:29:27.599414 22640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d19dd9e2d44227a7c2dd035c49f738, State: Initialized, Role: FOLLOWER
I20260620 12:29:27.599875 22640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d19dd9e2d44227a7c2dd035c49f738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33261 } }
I20260620 12:29:27.600032 22640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:29:27.600098 22640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:29:27.600194 22640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:27.601063 22640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d19dd9e2d44227a7c2dd035c49f738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33261 } }
I20260620 12:29:27.601405 22640 leader_election.cc:302] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91d19dd9e2d44227a7c2dd035c49f738; no voters:
I20260620 12:29:27.601727 22640 leader_election.cc:288] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:29:27.601955 22650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:27.602268 22650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [term 1 LEADER]: Becoming Leader. State: Replica: 91d19dd9e2d44227a7c2dd035c49f738, State: Running, Role: LEADER
I20260620 12:29:27.602576 22650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d19dd9e2d44227a7c2dd035c49f738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33261 } }
I20260620 12:29:27.603005 22640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:27.604468 22651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91d19dd9e2d44227a7c2dd035c49f738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d19dd9e2d44227a7c2dd035c49f738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33261 } } }
I20260620 12:29:27.604526 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91d19dd9e2d44227a7c2dd035c49f738. Latest consensus state: current_term: 1 leader_uuid: "91d19dd9e2d44227a7c2dd035c49f738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d19dd9e2d44227a7c2dd035c49f738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33261 } } }
I20260620 12:29:27.604593 22651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:27.604629 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:27.604940 22661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:27.607921 22661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:27.613215 22661 catalog_manager.cc:1347] Generated new cluster ID: a18987cd50594b098e5b438680965a6c
I20260620 12:29:27.613282 22661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:27.633597 22661 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:27.633927 22661 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:27.639909 22661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91d19dd9e2d44227a7c2dd035c49f738: Generated new TSK 0
I20260620 12:29:27.640569 22661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:29:27.646384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22572
2026-06-20T12:29:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1396 ms)
[----------] 18 tests from AuthTokenIssuingTest (25348 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:27 dist-test-slave-bwpp krb5kdc[22671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:27 dist-test-slave-bwpp krb5kdc[22671](info): set up 2 sockets
Jun 20 12:29:27 dist-test-slave-bwpp krb5kdc[22671](info): commencing operation
krb5kdc: starting...
W20260620 12:29:28.686981 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:28 dist-test-slave-bwpp krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:28.805915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46275
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46275,127.0.98.61:37933,127.0.98.60:43649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:28.917793 22684 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:29:28.918048 22684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:28.918071 22684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:28.918107 22684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:28.922003 22684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:28.922055 22684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:28.922075 22684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:28.922091 22684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:28.922106 22684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:28.927196 22684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46275,127.0.98.61:37933,127.0.98.60:43649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:28.928092 22684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:28.929005 22684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:28.937333 22692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:28.937306 22690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:28.937471 22684 server_base.cc:1034] running on GCE node
W20260620 12:29:28.937327 22689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:28.938128 22684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:28.939240 22684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:28.940433 22684 hybrid_clock.cc:648] HybridClock initialized: now 1781958568940407 us; error 45 us; skew 500 ppm
Jun 20 12:29:28 dist-test-slave-bwpp krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:28.943264 22684 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:28.944139 22684 webserver.cc:460] Webserver started at http://127.0.98.62:34769/ using document root <none> and password file <none>
I20260620 12:29:28.944722 22684 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:28.944828 22684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:28.945062 22684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:28.947089 22684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "2adeccbc79f94529b2bd65c30f96bcb9"
format_stamp: "Formatted at 2026-06-20 12:29:28 on dist-test-slave-bwpp"
server_key: "76083aa3422df819d30422210d03887a"
server_key_iv: "42dd0b7878b3d077ccc8ff622bf4763d"
server_key_version: "encryptionkey@0"
I20260620 12:29:28.947642 22684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "2adeccbc79f94529b2bd65c30f96bcb9"
format_stamp: "Formatted at 2026-06-20 12:29:28 on dist-test-slave-bwpp"
server_key: "76083aa3422df819d30422210d03887a"
server_key_iv: "42dd0b7878b3d077ccc8ff622bf4763d"
server_key_version: "encryptionkey@0"
I20260620 12:29:28.951321 22684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:29:28.953866 22699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:28.955510 22684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:29:28.955615 22684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "2adeccbc79f94529b2bd65c30f96bcb9"
format_stamp: "Formatted at 2026-06-20 12:29:28 on dist-test-slave-bwpp"
server_key: "76083aa3422df819d30422210d03887a"
server_key_iv: "42dd0b7878b3d077ccc8ff622bf4763d"
server_key_version: "encryptionkey@0"
I20260620 12:29:28.955701 22684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:28.988086 22684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:28.991520 22684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:28.991698 22684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:28.998831 22684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46275
I20260620 12:29:28.999115 22751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46275 every 8 connection(s)
I20260620 12:29:28.999903 22684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:29.001776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22684
I20260620 12:29:29.002039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:29.002344 392 external_mini_cluster.cc:1428] Setting key 5c2210896807d233f92e080b2729a250
I20260620 12:29:29.002892 22752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:29.006570 22752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:29.021654 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.010014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.0.98.62:46275)
0620 12:29:29.010358 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.010369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.010413 (+ 44us) server_negotiation.cc:407] Connection header received
0620 12:29:29.010796 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.010800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.010979 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.011178 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.012027 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.012886 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.013934 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.014311 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.014495 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.016749 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.016755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.016770 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.016790 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.018538 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.019040 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.019046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.019048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.019152 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.019477 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.019480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.019482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.019838 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.019985 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.020342 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.020610 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260620 12:29:29.021654 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.008971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46275 (local address 127.0.0.1:38654)
0620 12:29:29.009496 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.009538 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.010458 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.011176 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.011181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.011612 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.011849 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.011862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.013020 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.013024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.013813 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.013823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.014467 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.014471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.014563 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.015184 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.015201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.016600 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.018674 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.018684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.018694 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.018918 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.019251 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.019255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.019257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.019368 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.019961 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.019967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.020107 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.020284 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.020660 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":106,"thread_start_us":165,"threads_started":1}
I20260620 12:29:29.024226 22752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:29.024972 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.024726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53008)
0620 12:29:29.024845 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.024910 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 12:29:29.025180 22701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
I20260620 12:29:29.025961 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.025751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53018)
0620 12:29:29.025869 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.025910 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 12:29:29.026098 22752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37933: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 12:29:29.029474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37933
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46275,127.0.98.61:37933,127.0.98.60:43649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:29:29.075337 22752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } attempt: 1
I20260620 12:29:29.076238 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.075926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53024)
0620 12:29:29.076090 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.076162 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 12:29:29.077033 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.076807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53026)
0620 12:29:29.076918 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.076974 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 12:29:29.077157 22752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37933: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
W20260620 12:29:29.139353 22759 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:29:29.139672 22759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:29.139755 22759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:29.139842 22759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:29.144088 22759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:29.144173 22759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:29.144215 22759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:29.144281 22759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:29.144318 22759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:29.145316 22752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } attempt: 2
I20260620 12:29:29.146140 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.145847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53040)
0620 12:29:29.146000 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.146069 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 12:29:29.146858 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.146628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53056)
0620 12:29:29.146747 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.146810 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 12:29:29.147014 22752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37933: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37933: connect: Connection refused (error 111)
I20260620 12:29:29.149828 22759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46275,127.0.98.61:37933,127.0.98.60:43649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:29.150997 22759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:29.152015 22759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:29.161214 22765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:29.161214 22767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:29.161227 22764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:29.161634 22759 server_base.cc:1034] running on GCE node
I20260620 12:29:29.162194 22759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:29.163280 22759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:29.164446 22759 hybrid_clock.cc:648] HybridClock initialized: now 1781958569164416 us; error 44 us; skew 500 ppm
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:29.167213 22759 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 12:29:29.168102 22759 webserver.cc:460] Webserver started at http://127.0.98.61:42353/ using document root <none> and password file <none>
I20260620 12:29:29.168643 22759 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:29.168740 22759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:29.168972 22759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:29.171007 22759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data/instance:
uuid: "d1177ae8295a4fa08e493b45155766e8"
format_stamp: "Formatted at 2026-06-20 12:29:29 on dist-test-slave-bwpp"
server_key: "b94cc4841a2572d9be50e1dadde895ea"
server_key_iv: "19c88074ece35f66fbf57f57cbfad7df"
server_key_version: "encryptionkey@0"
I20260620 12:29:29.171557 22759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/wal/instance:
uuid: "d1177ae8295a4fa08e493b45155766e8"
format_stamp: "Formatted at 2026-06-20 12:29:29 on dist-test-slave-bwpp"
server_key: "b94cc4841a2572d9be50e1dadde895ea"
server_key_iv: "19c88074ece35f66fbf57f57cbfad7df"
server_key_version: "encryptionkey@0"
I20260620 12:29:29.175104 22759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:29:29.177342 22774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:29.178508 22759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:29.178711 22759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/wal
uuid: "d1177ae8295a4fa08e493b45155766e8"
format_stamp: "Formatted at 2026-06-20 12:29:29 on dist-test-slave-bwpp"
server_key: "b94cc4841a2572d9be50e1dadde895ea"
server_key_iv: "19c88074ece35f66fbf57f57cbfad7df"
server_key_version: "encryptionkey@0"
I20260620 12:29:29.178834 22759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:29.195438 22759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:29.198513 22759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:29.198859 22759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:29.205848 22759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37933
I20260620 12:29:29.206133 22826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37933 every 8 connection(s)
I20260620 12:29:29.206959 22759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:29:29.209651 22827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:29.213351 22827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:29.215582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22759
I20260620 12:29:29.215701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-1/wal/instance
I20260620 12:29:29.216006 392 external_mini_cluster.cc:1428] Setting key 9366eeae300f58f3947acbf0f7c2bfc0
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 12:29:29.227010 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.215243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.0.98.62:46275)
0620 12:29:29.215410 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.215414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.216453 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 12:29:29.217392 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.217396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.217452 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.217568 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.218711 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.219233 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.220455 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.220689 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.220818 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.223307 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.223313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.223319 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.223343 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.224735 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.225270 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.225276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.225280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.225384 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.225762 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.225766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.225767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.225957 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.226076 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.226769 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.226887 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:29:29.228060 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.215719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46275 (local address 127.0.0.1:38670)
0620 12:29:29.216301 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.216343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.217181 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.217733 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.217740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.218073 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.218534 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.218546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.219380 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.219386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.220325 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.220334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.220845 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.220849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.221024 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.221662 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.221677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.223128 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.224860 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.224866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.224878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.225126 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.225486 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.225490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.225492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.225626 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.226094 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.226100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.226420 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.226638 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.227014 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":163,"threads_started":1}
I20260620 12:29:29.229434 22827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:29:29.238197 22752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } attempt: 3
I20260620 12:29:29.240299 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.230002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53068)
0620 12:29:29.230142 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.230153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.230208 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.230824 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.230827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.230991 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.231057 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.231062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.232018 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.232021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.232538 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.232545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.233126 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.233129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.233214 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.233733 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.233743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.236472 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.238617 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.238622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.238625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.238951 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.239309 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.239312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.239314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.239412 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.239911 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.239914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.240004 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.240088 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.240182 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 12:29:29.240525 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.230126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.0.98.61:37933)
0620 12:29:29.230458 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.230467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.230483 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:29:29.230513 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.230517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.230706 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.230835 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.231165 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.231902 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.232648 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.233014 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.233132 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.236622 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.236638 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.236651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.236671 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.238490 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.239075 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.239081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.239082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.239169 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.239515 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.239519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.239520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.239761 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.239938 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.240204 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.240348 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 12:29:29.243571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43649
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46275,127.0.98.61:37933,127.0.98.60:43649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:29:29.243633 22827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:29:29.244464 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.244092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:56920)
0620 12:29:29.244290 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.244373 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 12:29:29.244671 22775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
I20260620 12:29:29.245954 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.245684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:56930)
0620 12:29:29.245832 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.245885 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 12:29:29.246102 22827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
I20260620 12:29:29.248262 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.238877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53076)
0620 12:29:29.239014 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.239032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.239115 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.239433 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.239439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.239616 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.239709 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.239715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.240799 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.240805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.241731 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.241742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.242316 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.242319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.242411 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.242973 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.242988 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.244457 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.246694 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.246700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.246705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.246913 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.247238 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.247241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.247243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.247331 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.247726 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.247729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.247838 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.247960 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.248104 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 12:29:29.248277 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.238901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.98.61:37933)
0620 12:29:29.239169 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.239177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.239190 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:29:29.239225 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.239228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.239274 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.239393 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.239835 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.240639 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.241864 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.242188 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.242291 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.244688 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.244691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.244693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.244714 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.246547 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.247030 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.247036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.247037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.247127 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.247435 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.247438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.247440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.247623 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.247736 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.248021 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.248129 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260620 12:29:29.249111 22752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:29.249840 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.249584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:56932)
0620 12:29:29.249706 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.249766 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 12:29:29.250806 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.250538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:56940)
0620 12:29:29.250704 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.250749 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 12:29:29.250973 22752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
I20260620 12:29:29.282135 22752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } attempt: 1
I20260620 12:29:29.283007 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.282690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:56956)
0620 12:29:29.282862 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.282926 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 12:29:29.283779 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.283555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:56972)
0620 12:29:29.283673 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.283723 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 12:29:29.283910 22752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
I20260620 12:29:29.295384 22827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } attempt: 1
I20260620 12:29:29.296190 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.295880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:56988)
0620 12:29:29.296039 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.296107 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 12:29:29.296949 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.296702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57004)
0620 12:29:29.296830 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.296884 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 12:29:29.297111 22827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
W20260620 12:29:29.356254 22834 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:29:29.356532 22834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:29.356555 22834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:29.356669 22834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 12:29:29.359071 22752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } attempt: 2
I20260620 12:29:29.359912 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.359610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57016)
0620 12:29:29.359767 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.359831 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 12:29:29.360647 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.360427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57018)
0620 12:29:29.360544 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.360589 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 12:29:29.360776 22752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
W20260620 12:29:29.361153 22834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:29.361232 22834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:29.361279 22834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:29.361316 22834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:29.361352 22834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:29.365301 22827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } attempt: 2
I20260620 12:29:29.366003 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.365719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57032)
0620 12:29:29.365871 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.365931 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 12:29:29.366721 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.366477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57040)
0620 12:29:29.366589 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.366638 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 12:29:29.366879 22827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43649: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43649: connect: Connection refused (error 111)
I20260620 12:29:29.367074 22834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46275,127.0.98.61:37933,127.0.98.60:43649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:29.367997 22834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:29.368973 22834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:29.377061 22841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:29.377036 22840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:29.377053 22843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:29.377460 22834 server_base.cc:1034] running on GCE node
I20260620 12:29:29.378057 22834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:29.379164 22834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:29.380354 22834 hybrid_clock.cc:648] HybridClock initialized: now 1781958569380337 us; error 32 us; skew 500 ppm
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:29.383314 22834 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 12:29:29.384145 22834 webserver.cc:460] Webserver started at http://127.0.98.60:43827/ using document root <none> and password file <none>
I20260620 12:29:29.384688 22834 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:29.384784 22834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:29.385028 22834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:29.387061 22834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data/instance:
uuid: "b6aae0bdfcb44b90b0366e5f31656b7f"
format_stamp: "Formatted at 2026-06-20 12:29:29 on dist-test-slave-bwpp"
server_key: "a15a7bf70ca7b325ed61a9f1fb49604a"
server_key_iv: "897aacd7294abaf032a7d6f6630e33c9"
server_key_version: "encryptionkey@0"
I20260620 12:29:29.387626 22834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/wal/instance:
uuid: "b6aae0bdfcb44b90b0366e5f31656b7f"
format_stamp: "Formatted at 2026-06-20 12:29:29 on dist-test-slave-bwpp"
server_key: "a15a7bf70ca7b325ed61a9f1fb49604a"
server_key_iv: "897aacd7294abaf032a7d6f6630e33c9"
server_key_version: "encryptionkey@0"
I20260620 12:29:29.391237 22834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:29:29.393527 22850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:29.394644 22834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:29.394866 22834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/wal
uuid: "b6aae0bdfcb44b90b0366e5f31656b7f"
format_stamp: "Formatted at 2026-06-20 12:29:29 on dist-test-slave-bwpp"
server_key: "a15a7bf70ca7b325ed61a9f1fb49604a"
server_key_iv: "897aacd7294abaf032a7d6f6630e33c9"
server_key_version: "encryptionkey@0"
I20260620 12:29:29.394994 22834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:29.413406 22834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:29.416314 22834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:29.416536 22834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:29.423496 22834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43649
I20260620 12:29:29.423782 22902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43649 every 8 connection(s)
I20260620 12:29:29.424500 22834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:29:29.427274 22903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:29.430434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 22834
I20260620 12:29:29.430550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781958290914605-392-0/minicluster-data/master-2/wal/instance
I20260620 12:29:29.430876 392 external_mini_cluster.cc:1428] Setting key 8b7051dd268d990fc74b83dbd1634a60
I20260620 12:29:29.431141 22903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:29:29.443735 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.433101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.98.62:46275)
0620 12:29:29.433228 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.433232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.433517 (+ 285us) server_negotiation.cc:407] Connection header received
0620 12:29:29.433899 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.433902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.433956 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.434054 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.434992 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.435480 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.436534 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.436795 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.436872 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.439632 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.439636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.439639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.439666 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.441383 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.441914 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.441918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.441920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.442003 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.442440 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.442443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.442445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.442693 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.442798 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.443417 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.443565 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:29:29.444571 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.432157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.98.60:43649)
0620 12:29:29.432563 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.432579 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.432610 (+ 31us) server_negotiation.cc:407] Connection header received
0620 12:29:29.433470 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.433493 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.433742 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.433934 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.434695 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.435758 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.436610 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.437140 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.437237 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.440130 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.440136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.440145 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.440165 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.441702 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.442151 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.442161 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.442163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.442249 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.442901 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.442904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.442908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.443206 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.443396 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.443622 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.443749 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":255,"thread_start_us":129,"threads_started":1}
I20260620 12:29:29.444571 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.432947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46275 (local address 127.0.0.1:38676)
0620 12:29:29.433349 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.433399 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.433677 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.434130 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.434135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.434510 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.434853 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.434866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.435592 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.435596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.436397 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.436406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.436934 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.436938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.437203 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.437975 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.437990 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.439478 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.441516 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.441524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.441533 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.441802 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.442100 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.442104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.442106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.442245 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.442814 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.442827 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.443130 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.443341 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.443658 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":62,"thread_start_us":103,"threads_started":1}
I20260620 12:29:29.445858 22903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:29.455080 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.446335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37933 (local address 127.0.0.1:53084)
0620 12:29:29.446490 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.446504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.446573 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.446857 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.446860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.447033 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.447153 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.447162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.448236 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.448240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.449062 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.449070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.449660 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.449663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.449733 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.450223 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.450237 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.451572 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.453522 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.453525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.453527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.453716 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.454035 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.454037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.454038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.454114 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.454644 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.454646 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.454777 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.454865 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.454963 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 12:29:29.455277 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.446477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.0.98.61:37933)
0620 12:29:29.446597 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.446600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.446611 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:29:29.446689 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.446695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.446749 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.446828 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.447275 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.448118 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.449176 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.449540 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.449640 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.451715 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.451723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.451728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.451767 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.453396 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.453824 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.453830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.453832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.453923 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.454247 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.454250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.454251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.454534 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.454640 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.454957 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.455087 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:29:29.455963 22903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:29.457154 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.446971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.0.98.62:46275)
0620 12:29:29.447073 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.447077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.447089 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:29:29.447178 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.447184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.447234 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.447352 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.448090 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.448851 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.449916 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.450243 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.450363 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.453004 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.453008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.453011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.453037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.455111 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.455580 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.455583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.455585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.455676 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.455991 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.455996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.455998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.456222 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.456473 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.456946 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.457048 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:29:29.458042 22827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } attempt: 3
I20260620 12:29:29.459908 22752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } attempt: 3
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:29:29.464730 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.456443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57066)
0620 12:29:29.456608 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.456621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.456696 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.456916 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.456919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.457076 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.457133 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.457137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.458073 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.458079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.458784 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.458793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.459385 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.459388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.459460 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.459862 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.459882 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.461333 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.463322 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.463325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.463327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.463553 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.463805 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.463808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.463810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.463897 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.464261 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.464264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.464341 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.464444 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.464583 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 12:29:29.465278 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.456520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.0.98.60:43649)
0620 12:29:29.456609 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.456614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.456662 (+ 48us) server_negotiation.cc:407] Connection header received
0620 12:29:29.456761 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.456765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.456797 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.456895 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.457456 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.457940 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.458921 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.459270 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.459655 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.461466 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.461470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.461472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.461491 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.463220 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.463647 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.463649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.463650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.463704 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.463997 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.464000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.464001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.464156 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.464261 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.465011 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.465110 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 12:29:29 dist-test-slave-bwpp krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:29:29.469033 22903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: Bootstrap starting.
I20260620 12:29:29.470611 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.458507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.0.98.61:37933)
0620 12:29:29.458761 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.458765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.458778 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:29:29.458823 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.458825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.458867 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.458930 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.459688 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.460553 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.461582 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.461893 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.461971 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.467200 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.467203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.467206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.467232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.468860 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.469346 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.469349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.469351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.469433 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.469700 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.469703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.469705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.469927 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.470152 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.470331 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.470459 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260620 12:29:29.470940 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.460266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57078)
0620 12:29:29.460530 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.460543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.460609 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.461286 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.461290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.461460 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.461555 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.461562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.463569 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.463572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.464450 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.464461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.465264 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.465268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.465381 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.465851 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.465865 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.467314 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.469345 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.469348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.469350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.469588 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.469901 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.469905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.469907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.470022 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.470437 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.470440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.470527 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.470604 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.470770 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260620 12:29:29.471453 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.460761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.0.98.60:43649)
0620 12:29:29.461060 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.461069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.461083 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:29:29.461115 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.461118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.461154 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.461668 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.462142 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.463444 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.464570 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.465138 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.465577 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.467460 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.467465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.467469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.467495 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.469241 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.469707 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.469710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.469712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.469792 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.470127 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.470130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.470131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.470321 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.471110 (+ 789us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.471180 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.471289 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260620 12:29:29.473768 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:29.474120 22752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9: Bootstrap starting.
I20260620 12:29:29.475023 22903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:29.475910 22903 log.cc:824] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:29.476644 22752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:29.476740 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.458384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43649 (local address 127.0.0.1:57068)
0620 12:29:29.458484 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:29.458497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:29.458571 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:29.459301 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:29.459306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:29.459480 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:29.459581 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.459590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.461378 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.461382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.462163 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:29.462171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.462749 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.462753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:29.462833 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.463264 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:29.463277 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:29.464662 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:29.475266 (+ 10604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.475272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.475274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.475471 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.475759 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:29.475762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:29.475764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:29.475850 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:29.476247 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:29.476251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:29.476343 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 12:29:29.476452 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 12:29:29.476587 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 12:29:29.476758 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.458454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.0.98.60:43649)
0620 12:29:29.459083 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.459090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.459100 (+ 10us) server_negotiation.cc:407] Connection header received
0620 12:29:29.459129 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.459131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.459167 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.459252 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.459706 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.461255 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.462298 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.462573 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.462745 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.464793 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.464797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.464799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.464826 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.475155 (+ 10329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.475570 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.475575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.475577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.475658 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.475940 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.475946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.475947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.476146 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.476243 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.476500 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.476602 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":94,"threads_started":1}
I20260620 12:29:29.477571 22752 log.cc:824] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:29.478055 22726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2adeccbc79f94529b2bd65c30f96bcb9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38684
I20260620 12:29:29.479485 22752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9: No bootstrap required, opened a new log
I20260620 12:29:29.479775 22827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: Bootstrap starting.
I20260620 12:29:29.481735 22827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:29.482389 22752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.482625 22752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adeccbc79f94529b2bd65c30f96bcb9, State: Initialized, Role: FOLLOWER
I20260620 12:29:29.482596 22827 log.cc:824] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:29.483048 22752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.483750 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.484045 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:29.484197 22903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: No bootstrap required, opened a new log
I20260620 12:29:29.484552 22752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:29.484612 22726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 12:29:29.484715 22726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:29.484769 22827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: No bootstrap required, opened a new log
I20260620 12:29:29.485863 22726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.486796 22726 leader_election.cc:288] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [CANDIDATE]: Term 1 election: Requested vote from peers d1177ae8295a4fa08e493b45155766e8 (127.0.98.61:37933), b6aae0bdfcb44b90b0366e5f31656b7f (127.0.98.60:43649)
I20260620 12:29:29.488799 22827 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.489001 22827 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1177ae8295a4fa08e493b45155766e8, State: Initialized, Role: FOLLOWER
I20260620 12:29:29.489300 22827 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.489362 22903 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.489599 22903 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6aae0bdfcb44b90b0366e5f31656b7f, State: Initialized, Role: FOLLOWER
W20260620 12:29:29.489651 22924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:29.489709 22924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:29.489880 22903 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.490339 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.490456 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:29.490617 22827 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:29.491058 22801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d1177ae8295a4fa08e493b45155766e8"
I20260620 12:29:29.491184 22903 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:29.491267 22801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:29.491298 22877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f"
I20260620 12:29:29.491525 22877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:29.492754 22801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adeccbc79f94529b2bd65c30f96bcb9 in term 1.
I20260620 12:29:29.492952 22877 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adeccbc79f94529b2bd65c30f96bcb9 in term 1.
I20260620 12:29:29.493225 22701 leader_election.cc:302] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adeccbc79f94529b2bd65c30f96bcb9, d1177ae8295a4fa08e493b45155766e8; no voters:
W20260620 12:29:29.493450 22705 tablet.cc:2369] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:29:29.496296 22913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:29.497224 22913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [term 1 LEADER]: Becoming Leader. State: Replica: 2adeccbc79f94529b2bd65c30f96bcb9, State: Running, Role: LEADER
I20260620 12:29:29.498636 22913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } }
I20260620 12:29:29.499622 22759 master_runner.cc:263] No leader master found from master d1177ae8295a4fa08e493b45155766e8
I20260620 12:29:29.499684 22759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:29:29.499892 22684 master_runner.cc:263] No leader master found from master 2adeccbc79f94529b2bd65c30f96bcb9
I20260620 12:29:29.499953 22684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:29:29.499971 22834 master_runner.cc:263] No leader master found from master b6aae0bdfcb44b90b0366e5f31656b7f
I20260620 12:29:29.500027 22834 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:29:29.501698 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adeccbc79f94529b2bd65c30f96bcb9. Latest consensus state: current_term: 1 leader_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.501866 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: This master's current role is: LEADER
W20260620 12:29:29.501924 22953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:29.501992 22953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:29.504304 22957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:29.505221 22957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 12:29:29.509581 22952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:29.509918 22952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:29.510246 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.510408 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:29.519845 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.502610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.0.98.61:37933)
0620 12:29:29.502740 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.502744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.506944 (+ 4200us) server_negotiation.cc:407] Connection header received
0620 12:29:29.507147 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.507151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.507196 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.507248 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.508682 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.509490 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.511201 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.511556 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.511667 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.513382 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.513386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.513389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.513413 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.515554 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.517722 (+ 2168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.517725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.517727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.517811 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.518538 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.518542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.518544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.518790 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.518885 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.519493 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.519641 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:29:29.521626 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.503864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.0.98.60:43649)
0620 12:29:29.503990 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.503994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.507786 (+ 3792us) server_negotiation.cc:407] Connection header received
0620 12:29:29.507972 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.507975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.508023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.508073 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.508905 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.509645 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.510757 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.511114 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.511199 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.513092 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.513095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.513098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.513124 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.516487 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.517307 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.517311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.517313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.517398 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.518114 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.518117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.518119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.518346 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.518464 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.521336 (+ 2872us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.521469 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:29:29.523940 22876 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [term 1 FOLLOWER]: Refusing update from remote peer 2adeccbc79f94529b2bd65c30f96bcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:29.523928 22801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [term 1 FOLLOWER]: Refusing update from remote peer 2adeccbc79f94529b2bd65c30f96bcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:29.524679 22913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:29:29.524858 22954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:29:29.528743 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adeccbc79f94529b2bd65c30f96bcb9. Latest consensus state: current_term: 1 leader_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.528853 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:29.529287 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adeccbc79f94529b2bd65c30f96bcb9. Latest consensus state: current_term: 1 leader_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.529407 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:29.529609 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:29.503456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.98.62:46275)
0620 12:29:29.503574 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.503578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.506933 (+ 3355us) server_negotiation.cc:407] Connection header received
0620 12:29:29.507147 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.507151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.507196 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.507248 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.511450 (+ 4202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.512201 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.523549 (+ 11348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:29.523866 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:29.523965 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:29.525727 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:29.525731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:29.525733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:29.525758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:29.527457 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.527961 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.527967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.527971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.528054 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:29.528353 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:29.528356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:29.528358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:29.528561 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:29.528688 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:29.529341 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 12:29:29.529458 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:29:29.532023 22954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.532147 22954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:29.532264 22962 mvcc.cc:205] Tried to move back new op lower bound from 7298902300761612288 to 7298902300675059712. Current Snapshot: MvccSnapshot[applied={T|T < 7298902300761612288}]
I20260620 12:29:29.533480 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.533609 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:29.534325 22964 mvcc.cc:205] Tried to move back new op lower bound from 7298902300761612288 to 7298902300675059712. Current Snapshot: MvccSnapshot[applied={T|T < 7298902300761612288}]
I20260620 12:29:29.541285 22957 catalog_manager.cc:1347] Generated new cluster ID: f1eeacc1cb7a4be481cfd9e93cf0f498
I20260620 12:29:29.541353 22957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:29.546118 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.546238 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:29.547295 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adeccbc79f94529b2bd65c30f96bcb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } peers { permanent_uuid: "d1177ae8295a4fa08e493b45155766e8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37933 } } peers { permanent_uuid: "b6aae0bdfcb44b90b0366e5f31656b7f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43649 } } }
I20260620 12:29:29.547405 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:29.548053 22963 mvcc.cc:205] Tried to move back new op lower bound from 7298902300761612288 to 7298902300675059712. Current Snapshot: MvccSnapshot[applied={T|T < 7298902300761612288}]
I20260620 12:29:29.563962 22957 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:29.567322 22957 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:29.580950 22957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2adeccbc79f94529b2bd65c30f96bcb9: Generated new TSK 0
I20260620 12:29:29.581833 22957 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:29:29.610478 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.607401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.0.98.61:37933)
0620 12:29:29.607539 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.607543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.609340 (+ 1797us) server_negotiation.cc:407] Connection header received
0620 12:29:29.609532 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.609535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.609585 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.609639 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.610330 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53122: BlockingRecv error: recv got EOF from 127.0.0.1:53122 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:29:29.611919 22756 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.608177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.98.62:46275)
0620 12:29:29.608304 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.608308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.611006 (+ 2698us) server_negotiation.cc:407] Connection header received
0620 12:29:29.611187 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.611190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.611249 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.611301 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.611804 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38706: BlockingRecv error: recv got EOF from 127.0.0.1:38706 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 12:29:29.613272 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.608785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.0.98.60:43649)
0620 12:29:29.608898 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.608902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.612381 (+ 3479us) server_negotiation.cc:407] Connection header received
0620 12:29:29.612544 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.612547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.612595 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.612643 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.613146 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57102: BlockingRecv error: recv got EOF from 127.0.0.1:57102 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 12:29:29.620688 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.619070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.0.98.61:37933)
0620 12:29:29.619194 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.619199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.619726 (+ 527us) server_negotiation.cc:407] Connection header received
0620 12:29:29.619949 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.619953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.620049 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.620100 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.620585 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53134: BlockingRecv error: recv got EOF from 127.0.0.1:53134 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 12:29:29.628999 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.627453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.0.98.61:37933)
0620 12:29:29.627555 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.627559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.628094 (+ 535us) server_negotiation.cc:407] Connection header received
0620 12:29:29.628295 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.628299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.628347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.628398 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.628890 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53148: BlockingRecv error: recv got EOF from 127.0.0.1:53148 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 12:29:29.634639 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.633154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.0.98.61:37933)
0620 12:29:29.633281 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.633285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.633774 (+ 489us) server_negotiation.cc:407] Connection header received
0620 12:29:29.633961 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.633964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.634011 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.634060 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.634543 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53162: BlockingRecv error: recv got EOF from 127.0.0.1:53162 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 12:29:29.642161 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.640681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.0.98.61:37933)
0620 12:29:29.640805 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.640810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.641278 (+ 468us) server_negotiation.cc:407] Connection header received
0620 12:29:29.641449 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.641452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.641511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.641562 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.642063 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53168: BlockingRecv error: recv got EOF from 127.0.0.1:53168 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 12:29:29.653962 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.652412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.0.98.61:37933)
0620 12:29:29.652525 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.652529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.653044 (+ 515us) server_negotiation.cc:407] Connection header received
0620 12:29:29.653248 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.653251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.653305 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.653355 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.653858 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53184: BlockingRecv error: recv got EOF from 127.0.0.1:53184 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 12:29:29.674368 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.672615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.0.98.61:37933)
0620 12:29:29.672740 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.672744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.673339 (+ 595us) server_negotiation.cc:407] Connection header received
0620 12:29:29.673554 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.673557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.673613 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.673669 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.674245 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53192: BlockingRecv error: recv got EOF from 127.0.0.1:53192 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 12:29:29.710251 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.708859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.98.61:37933)
0620 12:29:29.708993 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.708997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.709309 (+ 312us) server_negotiation.cc:407] Connection header received
0620 12:29:29.709460 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.709463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.709518 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.709618 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.710124 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53200: BlockingRecv error: recv got EOF from 127.0.0.1:53200 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 12:29:29.777768 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.776454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.0.98.61:37933)
0620 12:29:29.776642 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.776647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.776801 (+ 154us) server_negotiation.cc:407] Connection header received
0620 12:29:29.776990 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.776993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.777053 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.777154 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.777655 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53210: BlockingRecv error: recv got EOF from 127.0.0.1:53210 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 12:29:29.909371 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:29.908086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.0.98.61:37933)
0620 12:29:29.908281 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:29.908286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:29.908430 (+ 144us) server_negotiation.cc:407] Connection header received
0620 12:29:29.908610 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:29.908613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:29.908667 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:29.908770 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:29.909273 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53224: BlockingRecv error: recv got EOF from 127.0.0.1:53224 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 12:29:30.169095 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:30.167877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.0.98.61:37933)
0620 12:29:30.168041 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:30.168048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:30.168070 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:29:30.168312 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:30.168315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:30.168373 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:30.168478 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:30.168971 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53234: BlockingRecv error: recv got EOF from 127.0.0.1:53234 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:29:30.503698 22953 catalog_manager.cc:1259] Loaded cluster ID: f1eeacc1cb7a4be481cfd9e93cf0f498
I20260620 12:29:30.503775 22953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: loading cluster ID for follower catalog manager: success
I20260620 12:29:30.505427 22953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: acquiring CA information for follower catalog manager: success
I20260620 12:29:30.506853 22953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6aae0bdfcb44b90b0366e5f31656b7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:29:30.511150 22952 catalog_manager.cc:1259] Loaded cluster ID: f1eeacc1cb7a4be481cfd9e93cf0f498
I20260620 12:29:30.511236 22952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: loading cluster ID for follower catalog manager: success
I20260620 12:29:30.512737 22952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: acquiring CA information for follower catalog manager: success
I20260620 12:29:30.513808 22952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1177ae8295a4fa08e493b45155766e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:29:30.687749 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:30.683422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.0.98.61:37933)
0620 12:29:30.683655 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:30.683659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:30.683675 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:29:30.683797 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:30.683799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:30.683928 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:30.684042 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:30.684407 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.685075 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.686063 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.686336 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.686430 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:30.686897 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:30.687363 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:30.687460 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:30.687573 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 12:29:30.687622 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260620 12:29:30.695510 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:30.691445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.0.98.60:43649)
0620 12:29:30.691707 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:30.691715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:30.691771 (+ 56us) server_negotiation.cc:407] Connection header received
0620 12:29:30.691991 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:30.691994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:30.692119 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:30.692252 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:30.692522 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.693139 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.693998 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.694227 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.694343 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:30.694679 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:30.695114 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:30.695257 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:30.695297 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 12:29:30.695354 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260620 12:29:30.702850 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:30.698455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.0.98.61:37933)
0620 12:29:30.698572 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:30.698579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:30.699237 (+ 658us) server_negotiation.cc:407] Connection header received
0620 12:29:30.699355 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:30.699358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:30.699412 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:30.699491 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:30.699694 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.700341 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.701585 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.701901 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.701999 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:30.702284 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:30.702366 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:30.702413 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:30.702603 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 12:29:30.702699 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:29:30.703088 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:30.698873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.0.98.60:43649)
0620 12:29:30.699011 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:30.699015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:30.699271 (+ 256us) server_negotiation.cc:407] Connection header received
0620 12:29:30.699598 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:30.699603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:30.699648 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:30.699715 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:30.700047 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.700765 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.701817 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.702109 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.702218 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:30.702545 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:30.702615 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:30.702685 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:30.702906 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 12:29:30.702963 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:29:30.703385 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:30.698475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.98.62:46275)
0620 12:29:30.698768 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:30.698775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:30.698980 (+ 205us) server_negotiation.cc:407] Connection header received
0620 12:29:30.699088 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:30.699090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:30.699224 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:30.699357 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:30.699764 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.700432 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.701396 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:30.701702 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:30.701795 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:30.702233 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:30.702524 (+ 291us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:30.702642 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:30.702951 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 12:29:30.703020 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260620 12:29:30.706768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22684
I20260620 12:29:30.713606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22759
I20260620 12:29:30.719575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 22834
2026-06-20T12:29:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3074 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:29:30 dist-test-slave-bwpp krb5kdc[23052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:29:30 dist-test-slave-bwpp krb5kdc[23052](info): set up 2 sockets
Jun 20 12:29:30 dist-test-slave-bwpp krb5kdc[23052](info): commencing operation
krb5kdc: starting...
W20260620 12:29:31.767108 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/test-user.keytab.
Jun 20 12:29:31 dist-test-slave-bwpp krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:29:31.885327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46017
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46017,127.0.98.61:42561,127.0.98.60:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:31.993800 23065 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:29:31.994112 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:31.994184 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:31.994268 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:31.998136 23065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:31.998251 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:31.998301 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:31.998339 23065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:31.998375 23065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:32.003420 23065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46017,127.0.98.61:42561,127.0.98.60:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:32.004390 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:32.005342 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:32.013429 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:32.013474 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:32.013552 23065 server_base.cc:1034] running on GCE node
W20260620 12:29:32.013717 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:32.014297 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:32.015383 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:32.016566 23065 hybrid_clock.cc:648] HybridClock initialized: now 1781958572016545 us; error 32 us; skew 500 ppm
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:32.024086 23065 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:29:32.025005 23065 webserver.cc:460] Webserver started at http://127.0.98.62:46873/ using document root <none> and password file <none>
I20260620 12:29:32.025552 23065 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:32.025635 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:32.025856 23065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:32.027872 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data/instance:
uuid: "c9c406683fd741f7872c90c0dabef246"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "d3c8f3a6ee93fcb47937efa7e1e10f64"
server_key_iv: "441ee7d5b9c252f1b901f04fcf3b2fdc"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.028432 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9c406683fd741f7872c90c0dabef246"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "d3c8f3a6ee93fcb47937efa7e1e10f64"
server_key_iv: "441ee7d5b9c252f1b901f04fcf3b2fdc"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.031965 23065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:29:32.034250 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:32.035450 23065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:32.035588 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/wal
uuid: "c9c406683fd741f7872c90c0dabef246"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "d3c8f3a6ee93fcb47937efa7e1e10f64"
server_key_iv: "441ee7d5b9c252f1b901f04fcf3b2fdc"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.035696 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:32.049273 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:32.052017 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:32.052201 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:32.058849 23065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46017
I20260620 12:29:32.059144 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46017 every 8 connection(s)
I20260620 12:29:32.059749 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:29:32.060668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 23065
I20260620 12:29:32.060796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-0/wal/instance
I20260620 12:29:32.061122 392 external_mini_cluster.cc:1428] Setting key f9e2d98cc4b9d69e531dc58dcbcb254e
I20260620 12:29:32.063021 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:32.066639 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:32.082330 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.068865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.0.98.62:46017)
0620 12:29:32.069322 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.069341 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.069549 (+ 208us) server_negotiation.cc:407] Connection header received
0620 12:29:32.070482 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.070486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.070816 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.070999 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.071817 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.073002 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.074486 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.074892 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.075018 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.077610 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.077615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.077624 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.077643 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.079309 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.079758 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.079764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.079766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.079867 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.080183 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.080186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.080188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.080585 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.080721 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.081082 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.081315 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"spinlock_wait_cycles":60672,"thread_start_us":157,"threads_started":1}
I20260620 12:29:32.082330 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.068865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46017 (local address 127.0.0.1:40378)
0620 12:29:32.069299 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.069451 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.070214 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.071036 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.071045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.071430 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.071663 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.071676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.073140 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.073143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.074353 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.074369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.075063 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.075068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.075222 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.075804 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.075833 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.077464 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.079414 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.079424 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.079433 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.079650 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.079978 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.079981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.079983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.080090 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.080701 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.080707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.080846 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.081012 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.081392 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":100,"thread_start_us":105,"threads_started":1}
I20260620 12:29:32.084631 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:32.085350 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.085117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44810)
0620 12:29:32.085225 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.085292 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 12:29:32.085534 23084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 12:29:32.086485 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.086203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44822)
0620 12:29:32.086327 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.086388 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 12:29:32.087181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42561
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46017,127.0.98.61:42561,127.0.98.60:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:29:32.088454 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42561: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
I20260620 12:29:32.137722 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } attempt: 1
I20260620 12:29:32.138700 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.138356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44828)
0620 12:29:32.138516 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.138583 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 12:29:32.139439 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.139230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44832)
0620 12:29:32.139331 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.139376 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:29:32.139611 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42561: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
W20260620 12:29:32.198369 23140 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:29:32.198709 23140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:32.198769 23140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:32.198856 23140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:29:32.202869 23140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:32.202925 23140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:32.202975 23140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:32.203018 23140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:32.203061 23140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:32.207808 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } attempt: 2
I20260620 12:29:32.208417 23140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46017,127.0.98.61:42561,127.0.98.60:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:32.208652 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.208319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44844)
0620 12:29:32.208463 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.208531 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 12:29:32.209353 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.209128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44854)
0620 12:29:32.209239 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.209287 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:29:32.209529 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42561: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42561: connect: Connection refused (error 111)
I20260620 12:29:32.209527 23140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:32.210625 23140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:32.218847 23145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:32.218972 23140 server_base.cc:1034] running on GCE node
W20260620 12:29:32.218890 23146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:32.218865 23148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:32.219754 23140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:32.220832 23140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:32.221987 23140 hybrid_clock.cc:648] HybridClock initialized: now 1781958572221955 us; error 46 us; skew 500 ppm
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:32.224742 23140 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 12:29:32.225605 23140 webserver.cc:460] Webserver started at http://127.0.98.61:35741/ using document root <none> and password file <none>
I20260620 12:29:32.226158 23140 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:32.226246 23140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:32.226725 23140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:32.228778 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data/instance:
uuid: "3b7308eb992742c583fdeb93743a8964"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "8448ec9cc7d4622fecbcd271bf9595a9"
server_key_iv: "3b8e66577559762b090851fadb802c69"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.229331 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/wal/instance:
uuid: "3b7308eb992742c583fdeb93743a8964"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "8448ec9cc7d4622fecbcd271bf9595a9"
server_key_iv: "3b8e66577559762b090851fadb802c69"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.232908 23140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:29:32.235224 23155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:32.236428 23140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:29:32.236552 23140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/wal
uuid: "3b7308eb992742c583fdeb93743a8964"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "8448ec9cc7d4622fecbcd271bf9595a9"
server_key_iv: "3b8e66577559762b090851fadb802c69"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.236652 23140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:32.260049 23140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:32.263116 23140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:32.263396 23140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:32.270455 23140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42561
I20260620 12:29:32.270776 23207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42561 every 8 connection(s)
I20260620 12:29:32.271528 23140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:29:32.272933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 23140
I20260620 12:29:32.273062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-1/wal/instance
I20260620 12:29:32.273391 392 external_mini_cluster.cc:1428] Setting key ae62c6b6edfe4805c696f85b95bfbf83
I20260620 12:29:32.274741 23208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:32.278878 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:32.294373 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.280650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.98.62:46017)
0620 12:29:32.280834 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.280841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.281689 (+ 848us) server_negotiation.cc:407] Connection header received
0620 12:29:32.282938 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.282940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.282989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.283065 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.284240 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.285022 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.286913 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.287330 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.287451 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.290421 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.290426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.290433 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.290468 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.292221 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.292721 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.292726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.292728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.292829 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.293147 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.293150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.293153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.293372 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.293488 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.294103 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.294212 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:29:32.295523 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.280980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46017 (local address 127.0.0.1:40382)
0620 12:29:32.281511 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.281569 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.282597 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.283257 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.283265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.283614 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.284090 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.284103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.285159 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.285166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.286508 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.286519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.287525 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.287533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.287827 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.288521 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.288546 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.290246 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.292335 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.292339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.292352 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.292617 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.292933 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.292936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.292937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.293061 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.293492 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.293497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.293778 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.294014 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.294482 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":105,"threads_started":1}
I20260620 12:29:32.296950 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 12:29:32.300679 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } attempt: 3
I20260620 12:29:32.301455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
/tmp/dist-test-taskiQxjdC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33335
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46017,127.0.98.61:42561,127.0.98.60:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:29:32.307013 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.297449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44858)
0620 12:29:32.297556 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.297572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.297640 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.298156 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.298159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.298330 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.298396 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.298401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.299452 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.299455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.299966 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.299973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.300536 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.300539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.300662 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.301078 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.301092 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.302688 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.305115 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.305118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.305120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.305353 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.305682 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.305685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.305687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.305779 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.306303 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.306309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.306429 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.306562 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.306731 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":21}
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:29:32.307300 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.297550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.0.98.61:42561)
0620 12:29:32.297837 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.297847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.297864 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:29:32.297894 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.297898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.298044 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.298179 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.298525 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.299333 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.300074 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.300421 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.300529 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.302842 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.302854 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.302870 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.302902 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.305018 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.305474 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.305482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.305484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.305584 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.305882 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.305884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.305886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.306176 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.306322 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.306698 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.306862 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
I20260620 12:29:32.309710 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:32.310462 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.310208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53234)
0620 12:29:32.310316 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.310390 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 12:29:32.310690 23156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
I20260620 12:29:32.311486 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.311265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53244)
0620 12:29:32.311368 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.311422 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 12:29:32.311681 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
I20260620 12:29:32.312276 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.301315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44874)
0620 12:29:32.301491 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.301505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.301578 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.303152 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.303156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.303352 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.303450 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.303457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.304371 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.304375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.304908 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.304916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.305552 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.305555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.305674 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.306138 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.306153 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.307842 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.309833 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.309838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.309840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.310071 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.310407 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.310410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.310412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.310519 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.311884 (+ 1365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.311888 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.311978 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.312066 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.312165 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 12:29:32.312536 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.301338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.98.61:42561)
0620 12:29:32.302942 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.302949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.302960 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:29:32.302994 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.302996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.303028 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.303121 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.303560 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.304255 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.305029 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.305398 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.305538 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.308007 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.308011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.308013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.308041 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.309731 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.310162 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.310166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.310170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.310264 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.311476 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.311480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.311484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.311724 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.311894 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.312151 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.312280 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1526,"spinlock_wait_cycles":10496,"thread_start_us":80,"threads_started":1}
I20260620 12:29:32.313238 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:32.313872 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.313619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53248)
0620 12:29:32.313738 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.313789 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 12:29:32.314790 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.314468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53256)
0620 12:29:32.314598 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.314731 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 12:29:32.314917 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
I20260620 12:29:32.346107 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } attempt: 1
I20260620 12:29:32.347060 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.346748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53266)
0620 12:29:32.346913 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.346980 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 12:29:32.347874 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.347678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53270)
0620 12:29:32.347776 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.347820 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 12:29:32.348076 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
I20260620 12:29:32.360971 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } attempt: 1
I20260620 12:29:32.361845 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.361559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53278)
0620 12:29:32.361687 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.361761 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 12:29:32.362500 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.362304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53280)
0620 12:29:32.362404 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.362450 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 12:29:32.362622 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
W20260620 12:29:32.421635 23215 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:29:32.421959 23215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:29:32.422032 23215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:29:32.422122 23215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 12:29:32.423235 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } attempt: 2
I20260620 12:29:32.424057 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.423776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53296)
0620 12:29:32.423918 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.423985 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 12:29:32.424741 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.424532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53300)
0620 12:29:32.424636 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.424686 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 12:29:32.424865 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
W20260620 12:29:32.427251 23215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:29:32.427325 23215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:29:32.427376 23215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:29:32.427433 23215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:29:32.427474 23215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:29:32.430791 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } attempt: 2
I20260620 12:29:32.431449 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.431201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53302)
0620 12:29:32.431319 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.431375 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 12:29:32.432062 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.431865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53314)
0620 12:29:32.431956 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.432004 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 12:29:32.432173 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33335: connect: Connection refused (error 111)
I20260620 12:29:32.433472 23215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46017,127.0.98.61:42561,127.0.98.60:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:29:32.434578 23215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:29:32.435649 23215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:29:32.444173 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:32.444173 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:29:32.444211 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:29:32.444432 23215 server_base.cc:1034] running on GCE node
I20260620 12:29:32.445043 23215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:29:32.446061 23215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:29:32.447244 23215 hybrid_clock.cc:648] HybridClock initialized: now 1781958572447216 us; error 38 us; skew 500 ppm
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:29:32.450011 23215 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 12:29:32.450937 23215 webserver.cc:460] Webserver started at http://127.0.98.60:44869/ using document root <none> and password file <none>
I20260620 12:29:32.451526 23215 fs_manager.cc:359] Metadata directory not provided
I20260620 12:29:32.451618 23215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:29:32.451844 23215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:29:32.453908 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data/instance:
uuid: "007ad2f0e0bf47d09be3d8092851fee7"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "c5876dd1f7bdfd4ffea2f63880817491"
server_key_iv: "4b34991ca7e7c0fd6c285387870f3f6c"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.454494 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/wal/instance:
uuid: "007ad2f0e0bf47d09be3d8092851fee7"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "c5876dd1f7bdfd4ffea2f63880817491"
server_key_iv: "4b34991ca7e7c0fd6c285387870f3f6c"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.458321 23215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:29:32.460804 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:29:32.462001 23215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:29:32.462132 23215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/wal
uuid: "007ad2f0e0bf47d09be3d8092851fee7"
format_stamp: "Formatted at 2026-06-20 12:29:32 on dist-test-slave-bwpp"
server_key: "c5876dd1f7bdfd4ffea2f63880817491"
server_key_iv: "4b34991ca7e7c0fd6c285387870f3f6c"
server_key_version: "encryptionkey@0"
I20260620 12:29:32.462242 23215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:29:32.494876 23215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:29:32.497786 23215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:29:32.497993 23215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:29:32.505136 23215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33335
I20260620 12:29:32.505455 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33335 every 8 connection(s)
I20260620 12:29:32.506124 23215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:29:32.508570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu as pid 23215
I20260620 12:29:32.508701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiQxjdC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781958290914605-392-0/minicluster-data/master-2/wal/instance
I20260620 12:29:32.509042 392 external_mini_cluster.cc:1428] Setting key efad47fbdd97d765d488dc12aaab5ebb
I20260620 12:29:32.509042 23284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:29:32.513799 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:29:32.523329 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } attempt: 3
I20260620 12:29:32.523571 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.510294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.0.98.60:33335)
0620 12:29:32.510697 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.510720 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.510754 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:29:32.511686 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.511704 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.512026 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.512291 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.513160 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.514170 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.515351 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.515986 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.516179 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.519015 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.519020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.519029 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.519054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.520691 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.521107 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.521117 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.521126 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.521261 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.521565 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.521568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.521570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.522030 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.522234 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.522472 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.522786 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
I20260620 12:29:32.524029 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } attempt: 3
I20260620 12:29:32.525417 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.515612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.0.98.62:46017)
0620 12:29:32.515754 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.515759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.516026 (+ 267us) server_negotiation.cc:407] Connection header received
0620 12:29:32.516187 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.516191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.516238 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.516293 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.517166 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.517935 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.519162 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.519480 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.519594 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.521908 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.521911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.521913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.521934 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.523471 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.523975 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.523979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.523981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.524084 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.524435 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.524438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.524440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.524660 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.524752 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.525145 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.525256 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:29:32.526384 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.515539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46017 (local address 127.0.0.1:40384)
0620 12:29:32.515900 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.515948 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.516067 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.516492 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.516498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.516923 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.517042 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.517047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.518053 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.518057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.519057 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.519066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.519601 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.519604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.519684 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.520235 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.520247 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.521787 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.523567 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.523577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.523582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.523877 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.524191 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.524196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.524198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.524290 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.524799 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.524805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.524920 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.525774 (+ 854us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.525973 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":127,"threads_started":1}
I20260620 12:29:32.527604 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:29:32.535140 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.524395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53344)
0620 12:29:32.524565 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.524579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.524654 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.525418 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.525421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.525578 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.525672 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.525679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.526627 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.526630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.527492 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.527502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.528868 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.528872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.528960 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.529368 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.529381 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.531451 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.533556 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.533560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.533562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.533801 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.534116 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.534119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.534121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.534204 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.534643 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.534669 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.534756 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.534834 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.534955 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 12:29:32.535665 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.524861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.0.98.60:33335)
0620 12:29:32.525204 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.525207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.525218 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:29:32.525245 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.525250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.525291 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.525770 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.525841 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.526516 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.528447 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.528763 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.528865 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.531586 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.531592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.531598 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.531628 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.533431 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.533928 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.533931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.533936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.534014 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.534315 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.534318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.534319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.534529 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.535306 (+ 777us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.535391 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.535499 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":286,"thread_start_us":81,"threads_started":1}
I20260620 12:29:32.536296 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.526231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.0.98.62:46017)
0620 12:29:32.526328 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.526332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.526344 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:29:32.526498 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.526503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.526546 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.526604 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.527626 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.528300 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.529209 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.529541 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.529641 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.532486 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.532490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.532494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.532520 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.534135 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.534643 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.534677 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.534679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.534752 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.535065 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.535068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.535070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.535279 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.535457 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.536027 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.536131 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":941,"server-negotiator.queue_time_us":44}
I20260620 12:29:32.536476 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.523767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53336)
0620 12:29:32.524003 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.524018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.524095 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.525208 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.525213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.525411 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.525538 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.525545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.526831 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.526835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.527640 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.527648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.528664 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.528668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.528756 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.529406 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.529423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.531038 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.534278 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.534284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.534286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.534554 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.534885 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.534888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.534890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.534983 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.535936 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.535940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.536053 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.536147 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.536267 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 12:29:32.537211 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.524667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.0.98.60:33335)
0620 12:29:32.524957 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.524966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.524984 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:29:32.525019 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.525022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.525065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.525648 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.525734 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.526707 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.527769 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.528554 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.528643 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.531213 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.531219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.531224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.531255 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.534149 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.534705 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.534709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.534710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.534788 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.535336 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.535347 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.535351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.535583 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.536860 (+ 1277us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.536937 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.537044 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
I20260620 12:29:32.539424 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246: Bootstrap starting.
I20260620 12:29:32.539909 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.528093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.0.98.61:42561)
0620 12:29:32.528198 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.528202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.528215 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:29:32.528260 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.528262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.528300 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.528378 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.529700 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.530524 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.531667 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.532073 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.532158 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.534885 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.534888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.534891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.534931 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.536612 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.538386 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.538393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.538398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.538486 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.538935 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.538938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.538940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.539203 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.539273 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.539590 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.539714 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":2816}
I20260620 12:29:32.540055 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: Bootstrap starting.
I20260620 12:29:32.540352 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.528036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42561 (local address 127.0.0.1:44880)
0620 12:29:32.528121 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.528134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.528199 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.529247 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.529252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.529447 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.529564 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.529574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.530687 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.530690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.531549 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.531559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.532317 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.532321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.532442 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.532851 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.532870 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.534724 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.537997 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.538000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.538002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.538257 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.538674 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.538679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.538682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.538809 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.539345 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.539348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.539427 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.540082 (+ 655us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.540213 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 12:29:32.542107 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:32.542382 23208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: Neither blocks nor log segments found. Creating new log.
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:29:32.543206 23208 log.cc:824] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:32.543469 23133 log.cc:824] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:32.543608 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:29:32.545635 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: No bootstrap required, opened a new log
I20260620 12:29:32.545657 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246: No bootstrap required, opened a new log
I20260620 12:29:32.547760 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.537784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.0.98.61:42561)
0620 12:29:32.537909 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.537913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.537936 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:29:32.538001 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.538003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.538050 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.538103 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.539044 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.539757 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.540856 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.541096 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.541226 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.543869 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.543872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.543875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.543899 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.545623 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.546102 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.546105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.546107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.546184 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.546485 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.546488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.546490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.546729 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.546892 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.547503 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.547614 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":52}
Jun 20 12:29:32 dist-test-slave-bwpp krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:29:32.550043 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.550153 23208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.550305 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c406683fd741f7872c90c0dabef246, State: Initialized, Role: FOLLOWER
I20260620 12:29:32.550437 23208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7308eb992742c583fdeb93743a8964, State: Initialized, Role: FOLLOWER
I20260620 12:29:32.550843 23208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.550943 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:29:32.550901 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.552089 23107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c9c406683fd741f7872c90c0dabef246"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40392
I20260620 12:29:32.552459 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:32.552537 23107 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 12:29:32.552641 23208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:32.552659 23107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [term 0 FOLLOWER]: Advancing to term 1
W20260620 12:29:32.553589 23086 tablet.cc:2369] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:29:32.554054 23107 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.554772 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.544492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.0.98.60:33335)
0620 12:29:32.544579 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.544583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.544661 (+ 78us) server_negotiation.cc:407] Connection header received
0620 12:29:32.545068 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.545071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.545113 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.545159 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.545632 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.546343 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.547378 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.547693 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.547766 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.550539 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.550542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.550544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.550566 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.552278 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.552760 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.552763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.552765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.552847 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.553134 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.553136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.553138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.553339 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.553418 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.554396 (+ 978us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.554514 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:29:32.555095 23107 leader_election.cc:288] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [CANDIDATE]: Term 1 election: Requested vote from peers 3b7308eb992742c583fdeb93743a8964 (127.0.98.61:42561), 007ad2f0e0bf47d09be3d8092851fee7 (127.0.98.60:33335)
I20260620 12:29:32.555539 23182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9c406683fd741f7872c90c0dabef246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3b7308eb992742c583fdeb93743a8964"
I20260620 12:29:32.556939 23182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:32.558279 23182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c406683fd741f7872c90c0dabef246 in term 1.
W20260620 12:29:32.558408 23316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:32.558475 23316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:32.558681 23084 leader_election.cc:302] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7308eb992742c583fdeb93743a8964, c9c406683fd741f7872c90c0dabef246; no voters:
W20260620 12:29:32.559099 23311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:32.559213 23311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:32.559942 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.560125 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:32.560256 23294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:29:32.560369 23294 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [term 1 LEADER]: Becoming Leader. State: Replica: c9c406683fd741f7872c90c0dabef246, State: Running, Role: LEADER
I20260620 12:29:32.561136 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.544298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33335 (local address 127.0.0.1:53356)
0620 12:29:32.544607 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 12:29:32.544619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:29:32.544708 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:29:32.545267 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:29:32.545270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:29:32.545435 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:29:32.545518 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.545525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.546466 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.546469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.547253 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:29:32.547263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.547897 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.547900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:29:32.547990 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.548428 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:29:32.548443 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:29:32.550412 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:29:32.552394 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.552396 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.552398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.552636 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.552963 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:29:32.552966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:29:32.552968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:29:32.553045 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:29:32.554082 (+ 1037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:29:32.554088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:29:32.554198 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 12:29:32.554954 (+ 756us) client_negotiation.cc:240] Negotiation successful
0620 12:29:32.560978 (+ 6024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 12:29:32.563642 23294 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.564385 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: Bootstrap starting.
I20260620 12:29:32.567234 23284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: Neither blocks nor log segments found. Creating new log.
I20260620 12:29:32.568130 23284 log.cc:824] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: Log is configured to *not* fsync() on all Append() calls
I20260620 12:29:32.564175 23065 master_runner.cc:186] Error getting master registration for 127.0.98.60:33335: OK, instance_id { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" instance_seqno: 1781958572503101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:29:32.568570 23065 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:29:32.570246 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: No bootstrap required, opened a new log
I20260620 12:29:32.568537 23140 master_runner.cc:186] Error getting master registration for 127.0.98.60:33335: OK, instance_id { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" instance_seqno: 1781958572503101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:29:32.572341 23140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:29:32.572639 23323 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c406683fd741f7872c90c0dabef246. Latest consensus state: current_term: 1 leader_uuid: "c9c406683fd741f7872c90c0dabef246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.572755 23323 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:32.573056 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.573171 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:32.574393 23284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.574607 23284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007ad2f0e0bf47d09be3d8092851fee7, State: Initialized, Role: FOLLOWER
I20260620 12:29:32.575034 23284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } }
I20260620 12:29:32.576481 23284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:29:32.576540 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9c406683fd741f7872c90c0dabef246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "007ad2f0e0bf47d09be3d8092851fee7"
I20260620 12:29:32.576757 23258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:29:32.578593 23258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c406683fd741f7872c90c0dabef246 in term 1.
I20260620 12:29:32.582909 23330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.583112 23330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 12:29:32.583288 23341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:29:32.583393 23341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:29:32.583715 23327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:29:32.584805 23327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:29:32.590148 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.566861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.0.98.62:46017)
0620 12:29:32.566966 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.566971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.572581 (+ 5610us) server_negotiation.cc:407] Connection header received
0620 12:29:32.573094 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.573098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.573157 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.573209 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.580023 (+ 6814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.580708 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.582742 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.583065 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.583132 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.585664 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.585668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.585670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.585695 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.587609 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.588795 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.588798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.588800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.588882 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.589191 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.589194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.589196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.589407 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.589520 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.589873 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.590004 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:29:32.590515 23258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [term 1 FOLLOWER]: Refusing update from remote peer c9c406683fd741f7872c90c0dabef246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:32.591370 23323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:29:32.593859 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.566951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.0.98.61:42561)
0620 12:29:32.567061 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.567067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.575246 (+ 8179us) server_negotiation.cc:407] Connection header received
0620 12:29:32.575482 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.575486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.575532 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.575583 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.579436 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.580220 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.581329 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.581648 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.581723 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.586028 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.586032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.586034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.586060 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.588330 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.592585 (+ 4255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.592589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.592591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.592675 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.592959 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.592963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.592965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.593169 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.593271 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.593593 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.593705 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:29:32.595176 23330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c406683fd741f7872c90c0dabef246. Latest consensus state: current_term: 1 leader_uuid: "c9c406683fd741f7872c90c0dabef246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.595284 23330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:32.596227 23182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [term 1 FOLLOWER]: Refusing update from remote peer c9c406683fd741f7872c90c0dabef246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:29:32.596812 23323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:29:32.601262 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c406683fd741f7872c90c0dabef246. Latest consensus state: current_term: 1 leader_uuid: "c9c406683fd741f7872c90c0dabef246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.601390 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:32.605118 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9c406683fd741f7872c90c0dabef246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.605264 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:32.605655 23343 mvcc.cc:205] Tried to move back new op lower bound from 7298902313323380736 to 7298902313224892416. Current Snapshot: MvccSnapshot[applied={T|T < 7298902313323380736}]
I20260620 12:29:32.605662 23330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9c406683fd741f7872c90c0dabef246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.605777 23330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:32.606456 23344 mvcc.cc:205] Tried to move back new op lower bound from 7298902313323380736 to 7298902313224892416. Current Snapshot: MvccSnapshot[applied={T|T < 7298902313323380736}]
I20260620 12:29:32.612304 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:32.565986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.98.60:33335)
0620 12:29:32.566152 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.566157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.584466 (+ 18309us) server_negotiation.cc:407] Connection header received
0620 12:29:32.584621 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.584624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.584665 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.584712 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.585472 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.586893 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.590359 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:32.590712 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:32.590782 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:32.595506 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:29:32.595510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:29:32.595512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:29:32.595537 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:29:32.597221 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.610769 (+ 13548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.610774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.610776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.610902 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:29:32.611263 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:29:32.611267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:29:32.611269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:29:32.611505 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:29:32.611639 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:32.612000 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 12:29:32.612140 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 12:29:32.614208 23323 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9c406683fd741f7872c90c0dabef246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.614344 23323 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246 [sys.catalog]: This master's current role is: LEADER
I20260620 12:29:32.615511 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9c406683fd741f7872c90c0dabef246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c406683fd741f7872c90c0dabef246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46017 } } peers { permanent_uuid: "3b7308eb992742c583fdeb93743a8964" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42561 } } peers { permanent_uuid: "007ad2f0e0bf47d09be3d8092851fee7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33335 } } }
I20260620 12:29:32.615646 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:29:32.616228 23327 catalog_manager.cc:1347] Generated new cluster ID: 47d592323e224badb60213028acba1f0
I20260620 12:29:32.616293 23327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:29:32.641470 23327 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:29:32.642844 23327 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:29:32.658753 23327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9c406683fd741f7872c90c0dabef246: Generated new TSK 0
I20260620 12:29:32.659933 23327 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:29:32.689829 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.687253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.0.98.61:42561)
0620 12:29:32.687386 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.687391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.688739 (+ 1348us) server_negotiation.cc:407] Connection header received
0620 12:29:32.688955 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.688959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.689009 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.689064 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.689684 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44890: BlockingRecv error: recv got EOF from 127.0.0.1:44890 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 12:29:32.691525 23137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.688144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.0.98.62:46017)
0620 12:29:32.688261 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.688267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.690513 (+ 2246us) server_negotiation.cc:407] Connection header received
0620 12:29:32.690738 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.690742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.690794 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.690847 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.691381 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40418: BlockingRecv error: recv got EOF from 127.0.0.1:40418 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 12:29:32.693477 23289 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.692144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.0.98.60:33335)
0620 12:29:32.692310 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.692315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.692614 (+ 299us) server_negotiation.cc:407] Connection header received
0620 12:29:32.692776 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.692779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.692824 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.692872 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.693359 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53378: BlockingRecv error: recv got EOF from 127.0.0.1:53378 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 12:29:32.697966 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.696373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.98.61:42561)
0620 12:29:32.696493 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.696497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.697032 (+ 535us) server_negotiation.cc:407] Connection header received
0620 12:29:32.697243 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.697246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.697323 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.697375 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.697855 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44904: BlockingRecv error: recv got EOF from 127.0.0.1:44904 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 12:29:32.702749 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.701262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.0.98.61:42561)
0620 12:29:32.701356 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.701359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.701860 (+ 501us) server_negotiation.cc:407] Connection header received
0620 12:29:32.702052 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.702055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.702100 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.702150 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.702578 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44906: BlockingRecv error: recv got EOF from 127.0.0.1:44906 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 12:29:32.708105 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.706680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.98.61:42561)
0620 12:29:32.706773 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.706777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.707277 (+ 500us) server_negotiation.cc:407] Connection header received
0620 12:29:32.707445 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.707448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.707493 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.707542 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.708015 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44914: BlockingRecv error: recv got EOF from 127.0.0.1:44914 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 12:29:32.715406 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.713962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.0.98.61:42561)
0620 12:29:32.714082 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.714086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.714561 (+ 475us) server_negotiation.cc:407] Connection header received
0620 12:29:32.714755 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.714758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.714803 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.714851 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.715312 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44922: BlockingRecv error: recv got EOF from 127.0.0.1:44922 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 12:29:32.727042 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.725413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.0.98.61:42561)
0620 12:29:32.725519 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.725523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.726063 (+ 540us) server_negotiation.cc:407] Connection header received
0620 12:29:32.726246 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.726250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.726298 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.726348 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.726908 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44934: BlockingRecv error: recv got EOF from 127.0.0.1:44934 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 12:29:32.747575 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.745622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.0.98.61:42561)
0620 12:29:32.745749 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.745754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.746392 (+ 638us) server_negotiation.cc:407] Connection header received
0620 12:29:32.746628 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.746632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.746725 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.746787 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.747399 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44940: BlockingRecv error: recv got EOF from 127.0.0.1:44940 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 12:29:32.783442 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.782188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.61:42561)
0620 12:29:32.782315 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.782324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.782579 (+ 255us) server_negotiation.cc:407] Connection header received
0620 12:29:32.782700 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.782704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.782763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.782852 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.783323 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44954: BlockingRecv error: recv got EOF from 127.0.0.1:44954 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 12:29:32.850821 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.849570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.0.98.61:42561)
0620 12:29:32.849752 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.849757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.849902 (+ 145us) server_negotiation.cc:407] Connection header received
0620 12:29:32.850087 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.850094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.850150 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.850237 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.850711 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44960: BlockingRecv error: recv got EOF from 127.0.0.1:44960 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 12:29:32.982214 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:32.981001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.0.98.61:42561)
0620 12:29:32.981161 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:32.981166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:32.981300 (+ 134us) server_negotiation.cc:407] Connection header received
0620 12:29:32.981474 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:32.981478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:32.981536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:32.981627 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:32.982090 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44970: BlockingRecv error: recv got EOF from 127.0.0.1:44970 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:29:33.241551 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:29:33.240346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.98.61:42561)
0620 12:29:33.240496 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:33.240501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:33.240613 (+ 112us) server_negotiation.cc:407] Connection header received
0620 12:29:33.240802 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:33.240806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:33.240860 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:33.240948 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:29:33.241433 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44984: BlockingRecv error: recv got EOF from 127.0.0.1:44984 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:29:33.559803 23316 catalog_manager.cc:1259] Loaded cluster ID: 47d592323e224badb60213028acba1f0
I20260620 12:29:33.559885 23316 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: loading cluster ID for follower catalog manager: success
I20260620 12:29:33.561551 23316 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: acquiring CA information for follower catalog manager: success
I20260620 12:29:33.562728 23316 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b7308eb992742c583fdeb93743a8964: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:29:33.584650 23341 catalog_manager.cc:1259] Loaded cluster ID: 47d592323e224badb60213028acba1f0
I20260620 12:29:33.584733 23341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: loading cluster ID for follower catalog manager: success
I20260620 12:29:33.586310 23341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: acquiring CA information for follower catalog manager: success
I20260620 12:29:33.587437 23341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 007ad2f0e0bf47d09be3d8092851fee7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:29:33.759617 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:33.755889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.98.61:42561)
0620 12:29:33.756140 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:33.756143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:33.756213 (+ 70us) server_negotiation.cc:407] Connection header received
0620 12:29:33.756346 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:33.756349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:33.756484 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:33.756593 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:33.756818 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.757331 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.758210 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.758429 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.758507 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:33.758823 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:33.759265 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:33.759365 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:33.759445 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 12:29:33.759495 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260620 12:29:33.766345 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:33.762514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.0.98.60:33335)
0620 12:29:33.762807 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:33.762811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:33.762824 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:29:33.763004 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:33.763006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:33.763120 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:33.763220 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:33.763461 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.763954 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.764876 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.765141 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.765217 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:33.765534 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:33.765948 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:33.766070 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:33.766185 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 12:29:33.766232 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
I20260620 12:29:33.773104 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:33.769461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.0.98.61:42561)
0620 12:29:33.769588 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:33.769594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:33.769783 (+ 189us) server_negotiation.cc:407] Connection header received
0620 12:29:33.769839 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:33.769843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:33.769892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:33.769988 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:33.770207 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.770980 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.772041 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.772251 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.772361 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:33.772671 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:33.772749 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:33.772824 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:33.772918 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 12:29:33.772970 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:29:33.773674 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:33.769720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.0.98.60:33335)
0620 12:29:33.770136 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:33.770139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:33.770148 (+ 9us) server_negotiation.cc:407] Connection header received
0620 12:29:33.770205 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:33.770207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:33.770248 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:33.770321 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:33.770567 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.771243 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.772469 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.772777 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.772868 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:33.773206 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:33.773286 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:33.773375 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:33.773491 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 12:29:33.773537 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260620 12:29:33.774394 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:29:33.769613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.98.62:46017)
0620 12:29:33.769850 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 12:29:33.769857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:29:33.769875 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:29:33.770091 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:29:33.770095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:29:33.770252 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:29:33.770384 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:29:33.770747 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.771427 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.772611 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:29:33.772917 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:29:33.773010 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:29:33.773648 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:29:33.773979 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:29:33.774056 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:29:33.774166 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 12:29:33.774237 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260620 12:29:33.777066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 23065
I20260620 12:29:33.783566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 23140
I20260620 12:29:33.789270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiQxjdC/build/debug/bin/kudu with pid 23215
2026-06-20T12:29:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3069 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6144 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282867 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST